Just to highlight how unusual the [xx] spelling is, the Dictionary of the British English Spelling System (Greg Brooks, 2015) has this to say about the double-letter [xx] in English:
Doubled letter: (does not occur)
Indeed, [x] is like several other consonant letters that are "never or almost never written double": [h, j, q, v, w, & y] (p. 110). Only one exception is noted for [xx]: the brand name Exxon. (For another name, there's also the classic arcade game Zaxxon.) Anti-vaxxer, presumably, is too new or too slang to appear in this spelling dictionary. According to Merriam-Webster editor-at-large Peter Sokolowski, it was first introduced to the dictionary in 2018, and it was first used in 2009 [Snopes]. Understanding its early history may illustrate how the spelling developed.
Vax
Vax before the first usages of anti-vax was mainly an abbreviation used to brand specific vaccines (e.g. M-Vax, YF-VAX). I can find no instances of use outside of the medical community.
Anti-vax to Anti-vax(x)er
Selective search results for anti-vax on Google and JSTOR first yield results between 2005 and 2010. The initial usages occur in medical contexts as shorthand for people opposed to vaccinations:
The final group is those that are zealously anti-vaccination (anti-vax). (Stephen M. Perle and Randy Ferrance, "What's Good for the Goose Is ... Ethics and Vaccinations." *Dynamic Chiropractic, 23.4, 12 February 2005.)
The usage would slowly leak into the larger scientific and skeptic communities. By 2009, a skeptics conference called The Amazing Meeting had a session titled "Anti-Anti-Vax Panel: Steve Novella, Joe Albietz, Harriett Hall, Michael Goudeau" on its schedule (James Randi Educational Foundation). At around the same time, anti-vaxer and anti-vaxxer make appearances in the comment section to neurologist and skeptic Steven Novella's Neurologica blog post titled "A Personal Attack By J.B. Handley" (22 April 2009). Joseph writes:
Anti-vaxer anecdotes rarely make sense.
And cheglabratjoe writes:
Of course, creationism probably isn't going to directly kill anyone, so these antivaxxers are far more dangerous.
The use of both forms, together, within a day suggests that, early on, both forms were in use.
A post later that year from another skeptic group (Australian Skeptics) highlights how skeptics propagated the word: "The 2009 Richard Dawkins Award Goes to Bill Maher, Anti-Vaxer":
Decide for yourself, but we think the only way to fight the scourge of the anti-vaxers is to ensure they do not get a platform in the mainstream media.
Other comment sections from the period also attest to both forms being in use.
Anti-vaxxer
Between 2009 and today, the word has entered general discourse. It is used by both proponents and opponents of child vaccination. For some reason, anti-vaxxer became the preferred spelling, the one that Merriam-Webster selected and most editors went with.
The reason why is arbitrary. Editors and dictionaries usually like consistency, so they tend to choose one form. We could argue various formal explanations for why the form does or doesn't make sense, but it's hard to generalize from such an unusual form.
Finally, at the end of my process I found a Grammarphobia blog post that confirms the gist of this answer, while also including a meditation on why the [xx] spelling caught on over the [cc] spelling:
Why are the spellings “anti-vaxxer” and “anti-vax” more popular than “anti-vaccer” and “anti-vac”? Our guess is that English speakers prefer “xx” and “x” because it’s natural to pronounce them like the “cc” of “vaccine,” while “cc” and “c” could be pronounced like the “c” of “vacuum.”
