I heard it in The Big Lebowski movie, when Walter yells at Donny, "shut the fuck up, Donny"; then he exclaims, "You're out of your element!".
The references given in a comment (in thefreedictionary and urbandictionary) suggest the meaning, "to feel unhappy or strange because you are in a situation that you are not familiar with", but provide no sense of why it should mean that. It is a reference to classical elements, Earth, Water, Air, and Fire; as, for example, a fish out of water is out of its element.
Someone's "element" refers to their bailiwick of expertise or comfort. A doctor might be in her element in the operating room but out of her element in a garage.
The quote from The Big Lebowski can be interpreted as "This situation is not something you are comfortable or qualified to deal with."