Of course, as you mentioned, "gender" and "sex" can be synonyms.
The definition of the word "sex"
If you are describing the physical appearance of someone's body, the word "sex" is more likely to be applicable than "gender". The technical definitions of "sex" are more complicated than just chromosomes, though, so you aren't actually necessarily commenting on chromosomes when you use the word "sex".
After all, I'd assume you don't look at the chromosomes of these individuals to figure out their sex, right? Rather, humans can be "sexed" with high accuracy through observation of various anatomical features aside from their genitals because humans are a sexually dimorphic species. Of course, there are cases where people are mistaken about the sex of an individual: there's not really any way to eliminate that possibility 100% of the time.
I typed up a few more paragraphs about the biological criteria for determining the sex of an individual, but then deleted them because I doubt they're really relevant.
Knowing the definition of a word like "sex" isn't actually that useful
Legally, the definition of "sex" may be something else again, same for the definition of "gender" and any other terms. If you want to know how to CYA legally, you should talk to a law expert, not an English language expert. In the UK, the "Gender Recognition Act" seems to be one of the relevant laws when dealing with the concept of legal sex/gender.
You seem to allude in your question and comments to complications in using the words "sex" and "gender" in relation to transgender individuals: one thing that I would say you seem to be ignoring or unaware of is that a significant number (not all!) of transgender individuals don't particularly like it when other people bring up the subject of their sex. The term "gender dysphoria" refers to the distress that some individuals feel related to their sex or gender. People may be sensitive about statements that refer to their personal characteristics regardless of their accuracy. My understanding is that some transgender individuals may even feel distressed by references to sex as a binary categorization (even if the sex of those specific individuals is not referenced).
You cannot protect yourself from criticism (or legal prosecution, I imagine, although I am not a legal expert) just by making sure that everything you say is true. An example: the statement "So-and-so is Jewish" might be completely accurate, but still be criticized as anti-Semitic, depending on the context. This has nothing to do with the definition of the word "Jewish"; it's about the social implications of the statement, and how it affects people/how people react to it.