Edit based on the comment below: You should say "When I said 'you', I meant you in the general sense, not you specifically." Makes sense, since "general" is the opposite of "specific"! You could also use "generic" instead of "general" since you're using the generic "you" described below.
"You" is used as a collective pronoun when directed at a group of people. From Wikipedia:
a collective noun is a word used to define a group of objects, where objects can be people, animals, emotions, inanimate things, concepts, or other things. For example, in the phrase "a pride of lions," pride is a collective noun.
Barrie in the comments mentions "generic 'you'", which is an explicit or implicit use of "you" often used to refer to whomever might be around (or more often, reading) rather than being specifically directed at a group. (See http://en.wikipedia.org/wiki/Generic_you.)