There are people with far too high an opinion of their knowledge who criticize what they don't understand. That's the real explanation.
Of is a preposition; probably the most common one in English.
It's got thousands of special uses with special idioms
(Describe the meaning of of in the phrase the apple of my eye, for instance)
and having duties in possessive constructions with inanimate nouns (the arm of the chair),
and marking verbal arguments during nominalizations (the shooting of the hunters).
It's also a monosyllable /əv/ and it gets shortened often, since it's not stressed. Since there are only two phonemes in it, shortening can result either in a vowel /ə/ or a consonant /v/. Interestingly, these are also what the auxiliary verb have shortens into when it is unstressed, which is normally the case for auxiliary have of the perfect
- He would have gone. /hi'wʊdəv'ɡɔn/
leading to the classic misspelling (with identical pronunciation)
This is one place where people who pay too much attention to "correct" grammar get mistrustful of of. Of course it's silly, because people can't misspell what they're saying, but it's one way to put people down, and for too many people grammar is only good for putting people down.
Another way is to say that of is "unnecessary". This generally means that whoever has made the judgement does not understand what the function of the word is, and, noting that it can often be deleted, decides that it must always be deleted. And says so, and says that that is The Rule.
A good example is the phrase off of, normally pronounced /'ɔfə/, as in
- He had trouble keeping the cat off of the chair.
which can also be said without of:
- He had trouble keeping the cat off the chair.
The fact is that sometimes one wants an extra syllable and sometimes one doesn't,
and this optional of provides it when one needs it.
With quantifiers like all, each, any, some, many, much, few, etc. there are a lot of patterns,
- some with of required (all of us, but *all us),
- some with of optional (all the men, all of the men),
- some with of but a different number (each man, each of the men)
- and some without of (few houses, but *few of houses).
There are also interactions with articles and other prepositions. So it gets very complicated and there are lots of conflicting rules and that's when people get confused and start paying attention to opinions instead of facts. Since Anglophone schools no longer teach English grammar, and ESL classes always use their own teachers' versions of it, there is a great deal of confusion and attendant objection to almost everything about English grammar. Just consider the questions, comments, and answers on this site for examples.