As Huddleston & Pullum note, "you" can sometimes be used as a determiner:
You idiots never learn.
I'll never understand you idiots.
But this generally can't occur in the singular:
*You idiot never learn.
*I'll never understand you idiot.
But they mention that it can be used in the singular as part of a vocative:
You never learn, you idiot!
You idiot, I'll never understand you!
Is there a reason why the restriction to the plural exists everywhere except for the vocative? Or is this just a purely arbitrary rule?