I originally had this as a comment, but I think I'll expand it out.
Google's N-Gram viewer is an excellent opportunity to see not just how a word was used throughout history and how it is used now, but it can give an insight into the current trend to allow us to see how the gender association for a word might be down the road.
I'd say there's four general trends that we might see:
Single-gender consensus. For example, the word actress is undoubtedly femenine (he is/was/'s an actress registers exactly 0 hits). Adam's apple is going to be intensely associated as masculine in spite of being used occasionally for women.
Cross-gender consensus. For example, calling someone sexy is quite evenly distributed and has been since its introduction. Notice that I used several different forms of the phrase. When I first searched with just he/she is sexy, it swung in favor of women but still registering a quite sizable (33%ish) of masculine uses. Being completely unscientific and arbitrary, I'd sayif after including as many variations as feasible, if a word is in the 15-85% range for one gender's use (hence 85-15% for the other), we can conclude that it can be safely used for both genders.
Converging use. This would be a term that used to be heavily gendered, and is now seeing increased use for the other gender and, based on the trend lines, showing an increasing amount of parity in recent times. For example, with nurse or actor. While it may strike some or many as odd the use with either gender, that is the direction these words are going towards.
Divergent use. These are words that may have been less gendered in the past, but now are trending toward gender-exclusivity. While I'm sure there are others, the best one I can come up with is handsome. Since 2000 the female use has a marked uptick after a steady trend to obscurity, though the gains on the male side are larger. These are still usable for both genders, but maybe start to sound strange to some when used with a particular gender (I know most of my students find it quite odd to hear a woman called handsome).
Sometimes the graphs aren't perfect. For example, we can all agree (I hope) that wrestler can be appropriately used with each gender. That said, none of the N-Grams I tried could register a use with women — only men — until I tried male/female wrestler. Bingo, huge surge for female. Since clearly it was a male dominated term, we can see its close gender association from in the past, such that people felt the need to prefix female in front of it. The same happen with nurse where we can also see after the idea of male nurses caught on, there was a reduction in usage of the term with a continual increase in the use of female nurse.
Context of course is lost, so jocular vs serious uses might not be easily distinguishable (calling a guy vs gal "pretty" for me, at least, has vastly different connotations). But it at least gives some data to back up (or not) what might be found in dictionaries whose lexicographers presumably have already done much of the work to determine the gender associated with words in actual use.