"Feminist" -- by common definition, common usage, and by etymology -- means a focus on the rights of those who claim female gender identity, in the same way that "Environmentalist" is someone who cares mainly about environmental issues.
A perfectly valid interpretation is that a Feminist might be someone who believes in equal rights for all, even though their main interest or activity is in the domain of the rights of those with female gender identity.
It would seem obvious to me, but I suspect not to everyone, that at least a decent proportion of people who are interested in rights for women would have at least some interest in rights for others, too, since they've already demonstrated empathy by caring about a single set of rights.
However, the term implicitly excludes any significant interest in the rights for minorities based upon religion, race, age, sexuality and so forth. Basically lack of interest in any issues unrelated to gender identity. It also arguably explicitly excludes interest in the defense of rights of those with male gender identity, potentially including the transgendered.
There is considerable scope for argument about which terms are excluded, and whether the exclusion is explicit, implied, or imaginary. That is, the term's ambiguous about the claimant's beliefs, which can damage communication.
It seems clear though, that the term gives no explicit information about whether they do, or not, support rights for any other class of person. In this way, the term is again similar to "Environmentalist".
It also seems clear that (again like Environmentalist!) the term carries a whole lot of baggage for many people.
Is the term "wrong" to describe supporting universal equality?
English is usually more flexible than that.
There is no "wrong" unless you ask a prescriptivist. So this may not be the best question to ask, but perhaps we can find a question that answers the puzzlement behind the question.
Is the term "less than ideal" to describe supporting universal equality?
I would say, yes.
The facts that I am writing this now, that you are asking this question in the first place, and that you had the discussion that spawned the question, all imply that it was not the clearest possible term in that original discussion, since its meaning was not clear between two people using it in everyday speech. Its use likely derailed the conversation into a discussion of its meaning.
I would say that most people who do not self-identify as feminists, and are not well-read in modern feminist literature, will not ascribe the meaning of "pursuing universal equality" to someone who identifies as a feminist. They would, at best, say "pursuing equal rights for women".
So, for that meaning, it is certainly unclear. You would not say "Environmentalist" in this way, after all.
Are there clearer terms to use?
Well, let us consider those who campaign for equal rights based on things other than gender. How do they describe themselves?
Inclusive. Ecumenical. Fair trade. Democratic. Liberal. Socialist. Communist. Meritocrat. Social Justice campaigner. Occupier. Environmentalist. Rights activist. Human Rights supporter. Equal rights advocate. Pro-tolerance. Anti-discriminatory. Unprejudiced. And lots and lots more Pro-, Anti- and Un- ones.
Of these, some clearly leap out as comparable in their amount of focus: "Environmentalist" implies an interest in the environment, and implies nothing further about any other interests in rights.
"Rights Activist" or "Rights Campaigner" seems good, but carry extra meaning - they imply activism. Feminism is a belief, a stance, which requires no action to be taken.
In that case, the best term I can come up with to describe someone who holistically believes in equal rights for all is "Equal Rights Advocate", and for someone who is both that and who's main focus is feminism is the even-more-of-a-mouthful "advocate of equal rights and feminism" (in that order, otherwise it just sounds like a rephrased repetition of feminism for emphasis).
I am unhappy with both terms, and am sure that in both cases, there's a better one. But just "Feminism" on its own, isn't it.
[Edit: the further question asked was: can you be accused of misogyny for not ascribing this meaning to the word "Feminist"? It'd be unjust to do so, but people get very invested in their personal interpretations of their self-applied labels ("Christian", "Atheist", "Satanist", "Goth", "Geek", "Hacker"), and often gleefully take umbrage when people do not give those labels the same interpretation as their own in-crowd. So the answer is, if someone wants to pick a fight, they will pick a fight. Fight-picking over terminology is an excellent way of identifying people worth avoiding.]
I'm not here to debate the truth of what they do support but simply the semantics behind the word.
You probably can't separate the two. English is a common-usage language, so words mean whatever they're commonly used to mean. "Feminism" is an inherently politically charged word, so you just can't separate its "semantic" meaning from the meaning(s) it is commonly used to convey. My favorite example of this is the word "coffin." Once upon a time, it was simply a synonym for "box." Then people started using it as a euphemism for the box bodies get put in, and now that's all it means.