The term 'connotation' refers to the communication of a secondary meaning, hint, impression ... which is essentially dependent on both the word etc itself and the emotional / mental state of the hearer, their education, their place in society, their experiences ... So there is subjectiveness involved. The Dictionary.com definition, sense 2, and its accompanying example, bring out this duality (though it could be explained more clearly):
something suggested or implied by a word or thing, rather than being
explicitly named or described:
“Religion” has always had a negative connotation for me.
M-W gives definitions stressing (1) the purely objective and (2) both the objective and the subjective flavours involved (re-ordered & re-numbered):
(1) something suggested by a word or thing
(2) an idea or quality that a word makes you think about in addition to
CDO seems to add a further layer of complexity (associated thoughts ---> triggered reactions):
The word "lady" has connotations of refinement and excessive
femininity that some women find offensive.
If a word produces negative feelings with you, it has a negative connotation. That's not to say that it will have that effect on the vast majority of the population. Some words do, but some words have a negative connotation for just a handful of people. Denotations are far less subjective.
For a person who's had a messy divorce, even an innocuous 'ex-' may well trigger a strong negative response, though logically 'ex-serviceman' say might be expected to trigger more noble sentiments.
As an aside, one can easily end up treading on eggshells if one carries the fear of saying something potentially offensive or otherwise upsetting to someone to extremes. Some areas are obviously best avoided in almost all situations, and some in known temporary circumstances, but I've known the mention of the word 'pink' cause a (thankfully humorous) strong reaction in one person. You can't really find a workaround for 'X-ray'.