This is an interesting question! I can’t find any documentation specifically on the issue, so I don’t have a conclusive answer, but here anyway is a lot of relevant information, and a bit of further speculation.
The OED lists various pronunciations:
Brit. /mam/, /mɑːm/, U.S. /mæm/, /mɑm/; (unstressed) Brit. /məm/, U.S. /əm/, /mˌ/.
It also sheds some light on the recent historical usage:
The γ, δ, and ε forms [respectively mem, mim; mum, mom; and ’m] represent pronunciations formerly common in British regional usage and in the speech of domestic staff and others of similar status; such forms and pronunciations are also well attested in U.S. regional use, especially in yes ma’am (see yessum adv.) and no ma’am, and as the second element in school-marm n. Compare marm n.
Buckingham Palace protocol (c1990) directed that ‘the Queen should be addressed as “Ma’am” (to rhyme with jam).’
It also includes, later:
In 1936, R. W. Chapman ( S.P.E. Tract ii. 241) observed that ‘Except to royal persons, the contraction (whether mahm or măm) seems to be going out.’
This is all informative, but also a bit confusing. The main thing that’s clear is that the variety of pronunciations goes back quite a long way.
The second thing is that in modern BrE, the usage of ma’am is so restricted that it’s very hard to disentangle “what people now use” and “what Buckingham Palace asks for”. It’s a prescriptivist’s dream!
The third is that the OED itself seems a bit confused: the very pronunciation that the Palace asks for, /mæm/, is one which the OED lists only as U.S. usage.
The big question this leaves unresolved is: is it only recently that the Palace has asked for “jam”, or was this already officially preferred in the past?
Another factor which might be involved is how in many words, the vowel /a/ (of farm) is partly shifting to /æ/ (of ham) in prestige BrE accents, and the vowel of jam/*ham* in these accents has changed. In eg the early 20th century, in upper-class British accents, the vowel in ham, jam etc. was more raised and fronted than today, somewhat closer to hem. Conversely, words like glass, class were uniformly pronounced with the long ah vowel, /a/. In many lower-class accents, ham was much like today, with /æ/, and grass, etc. were also pronounced with this vowel.
Since then, the stigma of perceived lower-class and regional accents has decreased, whereas a stigma of being ‘too posh’ has become more widespread; so the raising/fronting of ham is now very rare, and the pronunciation of grass with the ham vowel /æ/ is somewhat more common among RP speakers.
Given that this shift involves many class issues, and the alternation of /a/ and /æ/, I suspect it may have influenced the pronunciation of ma’am somewhat. I’m pretty sure that the shift in the vowel of ham is relevant, given the older spellings of ma’am as mem, mim listed in the OED. The relevance of the change in grass is much more speculative.