The phrase is, "You are so retarded" -- the sentence calls for an adjective -- or "You are a retard". Either way, it is a short way of saying "mentally retarded", that is, suffering from a mental illness that results in abnormally low intelligence. When used outside a clinical setting, it means "You are stupid". This is generally considered not politically correct these days, as it makes light of mental illness. In some circles people might criticize you for using this phrase. It's also highly insulting. Depending on your relationship with your friends, this may be considered an inappropriate thing to say, or it might fall into the category of good-natured banter. If you told this same friend, "You are incredibly stupid", would he get mad or would he laugh? It would be in the same category.
Assuming that such an insult would be taken as a joke or a minor insult -- or that you want to insult your associate -- you might use such a phrase when the person has done something dumb or foolish.
"You are so lame" means that the person has said or done something inappropriate or awkward. We often say "That was so lame", referring to the action, rather than the person. Like, a man tries to ask a woman on a date and uses some cliche pick-up line. She brushes him off. His friends say, "That was the best pick-up line you could think of? That was so lame!" Or someone makes a poor excuse for failure. This might be labeled "lame".
While calling something "lame" might be considered to be making light of physical handicaps, it doesn't have the same negative PC connotations that "retard" does.