Imagine you are talking to a coterie:
Who will vote for Obama?
(2 hands raise)
I want to ask you two...
How many can follow up with "you" if there are more people to imply?
Who will vote for Obama?
(15 hands raise)
I want to ask you fifteen...
It sounds weird for me. I think the maximum number is 3, cause you four starts sounding strange for me. If it is true, how should I mention the number properly?
No limit; "I want to ask you fifteen thousand, three hundred and forty-seven" is grammatically sound.
It is a bit strange though, but consider the reason behind the phrasing. Picture the moment when the two people have raised their hands.
At that point in time, you're seeing two people, and counting to two in an instant, and so also are everyone present; both the two who raised their hands, and the rest who did not.
And all of that makes "you two" a useful means of identifying them, and making it clear that the "you" of your statement addresses them alone, rather than anyone else there.
The higher the number the less this is true. It's not strange as a feature of English, it's strange as a thing you'd want to say at all.
