English as she is spoke
It's important to remember that good English is more than just correct syntax and grammar. It is equally about clarity of expression, contextual understanding, customs, tropes, clichés, rhythm and sonic texture. It also evolves: almost nobody today speaks in the same way as their Victorian forebears.
Full of sound and furry
When we hear speech, we don't pay attention to individual words, but to the sounds they make. We understand meaning because we subconsciously pattern-match those sounds to existing chunks we've already stored away in our brains from countless prior conversations, each of which carries snippets of meaning both individually and also when combined. The more differentiated those sounds are, the less we have to think to derive meaning. Conversely, if we come across an uncommon, ambiguous, or unusual sound, it doesn't immediately register and we can experience a momentary discomfort while we regain our bearings. Fundamentally, for some people, the problem with 'who of you' is not grammar, but the noise it makes.
The funky gibbon
The two long 'o's in 'who' and 'you' are separated by a single short word also beginning with 'o'. Moreover, there are no hard consonants or sibilants to break up the phrase. When spoken therefore, the overall sonic effect is "oo-o-yoo", or perhaps something like "oo-a-yoo", depending on pronunciation. Apart from sounding like a monkey, this is a lot like the sound made by the question "who are you?", which is an extremely common sound-meme. This switching of the common translation of the sound for a different one may be the reason it feels a bit awkward, as Ricky suggests. For a brilliant example of this see the Two Ronnie's sketch 'Four Candles'.
Finally, returning to the question
Which of you watched Sherlock at Christmas?
Who watched Sherlock at Christmas?