The key word is context.
If you just say either of these two words, without context, many people may be confused as to which of the two you mean.
However, in context, this will rarely happen, because man is highly informal, both as a form of address, and certainly as an interjection, whereas ma'am quite formal.
For example, the interjection man would never be used together as ma'am like this:
Man, it's hot! Can I offer you a cold drink, ma'am?
When you use man to address someone, it's also informal:
You wanna cold beer, man?
* You wanna cold drink, ma'am?
Could I offer you a refreshment, ma'am?
* Could I offer you a refreshment, man?
The second and fourth variants are unlikely to be spoken by native speakers, because they mix different levels of formality. So in practice, the context will make the listener understand which of the two is meant.
As such, your confusion between the
m and
n sound is logical, the two
do sound very similar, even to native speakers, which can sometimes lead to confusion, for instance when saying, or even spelling a name over the phone.
Em and
en are so close that it often gets clarified when spelling out a word or name (M as in Mary, N as in Never, or the NATO-alphabet,
Mike versus
November).
Whenever words are used in context, confusion doesn't happen that often, though. For instance, the d and the b can sound very similar, but nobody will think that
The dog barked in the dark night.
actually is supposed to be
The bog darked in the bark night.
Even though bog and bark are existing words!