I would say it is a combination of two factors that show up separately in other sounds with token frequency on the low end.
/ʒ/ never developed in vocabulary inherited from Proto-Germanic
English is categorized as a Germanic language, which is a group of languages that are all thought to be descended from a common ancestor that we call Proto-Germanic. (The modern language called German is another one of these languages, but despite the name, Proto-Germanic has no special connection to German compared to English, Dutch, Swedish, or any other Germanic language.)
Many English words come from other sources, such as borrowing, but the inherited vocabulary—meaning vocabulary that was transmitted from Proto-Germanic continuously, without passing between separate languages—constitutes a large portion of the most frequent and basic words in English.
There is no regular source of /ʒ/ in vocabulary inherited from Proto-Germanic. This is also the case for /oɪ/, the least common vowel sound.
The sound /dʒ/, another relatively infrequent consonant sound, is partly similar in that it has no regular source in word-initial position in inherited English vocabulary. (Word-initial /dʒ/ occurs in borrowed words, and also in words whose etymology is unclear or involves irregular or sporadic sound changes.)
The main source of /ʒ/ is a sequence, /zj/
Even in borrowed vocabulary, /ʒ/ almost always originates historically from a sequence /zj/. (There are exceptions to this generalization, but they occur in a small number of words of relatively low frequency such as garage and genre, so they can be ignored for the sake of your question.) A sequence of sounds cannot be more frequent than either of its components, and so if the components were originally no more frequent than other single consonant sounds such as /n/, /l/, /v/, we would expect /zj/ (and therefore /ʒ/) to be less frequent than most single consonant sounds, and only as common as sequences like /nj/, /lj/, /vj/.
Furthermore, the /zj/ sequence that developed into /ʒ/ only occurred in the middle of words, so /ʒ/ is generally absent from monosyllables. It also doesn't occur in any frequent affixes (the /ɑːʒ/ pronunciation of -age is fairly marginal).
Other sounds in English that historically derive mostly from sequences, such as /ŋ/ (from the sequence /ng/, except for when it comes immediately before /g/ or /k/) or /ʃ/ (from *sk in native Germanic words, or from /sj/ in non-native Latinate words) are fairly low in frequency; however, these are probably not as infrequent as /ʒ/ because they do occur in native vocabulary, and they can occur in monosyllables (the *sk sequence that developed to /ʃ/ could occur word-initially or word-finally, and the /ng/ sequence that developed to /ŋ/ could occur word-finally). Furthermore, ruakh pointed out in the comments that /ŋ/ occurs in the common suffix -ing, which will raise its frequency. The phoneme /ʃ/ occurs in certain fairly common suffixes such as -ish and -tion.