Is "I'm screwed" a rude expression, or can it be used when someone tries to say they made a mistake?
I overheard it from someone who seemed to have failed at his task.
It's something you wouldn't say to your grandmother, perhaps, but it is a very common euphemism people use to avoid saying something stronger. It's used in everything from pop songs to commentaries by TV talk show hosts.
Seriously, in most walks of society screwed is considered merely informal these days, not rude.
Robusto's response addresses the rudeness aspect of your question.
With regard to the second part, whether I'm screwed is "used when someone tries to say they made a mistake": I think you're confusing I'm screwed (which as the comments tell you means approximately "Aw, jeez, I'm in trouble") with I screwed up, which does mean "I've made a mistake".
Another version of screwed up, made famous in The Right Stuff, is screwed the pooch; this is somewhat intensive, but the conventional intensifier is royal: "I screwed up royally."
Both expressions (and the "stronger" version to which Robusto alludes) may be used in second or third person as well as first, in either the singular or plural, and in any tense, aspect or mood.
Screw up is also frequently used adjectivally ("That's a really screwed-up piece of design") and nominally ("I'm getting tired of Carol's screw-ups"). The nominal version of screw in this sense is more often the gerund screwing than the infinitive screw (which tends to be reserved for the sexual sense); I cannot recall encountering an adjectival use.
I do believe that when you are saying "I'm screwed" or "He really screwed me" you are using a euphemism which has its roots in the sex act, which is sometimes referred to as "screwing". In this case, it is a less-vulgar version of saying "I'm fucked" or "He really fucked me". It is using the sex act as a metaphor for domination, which apparently is NEVER acceptable in polite company. Wiktionary says:
As for the term "screw the pooch", here's a quote from Jacob May, the DJ who first coined the term back in the '50s recounting how that came about:
So FtD and its nicer version, "screw the pooch" mean to screw up EXTREMELY BADLY, somewhat how FUBAR means something ""f'd" up beyond all reason" Worse than we can describe in words. So bad that it would be as big a screw up as having intercourse with a dog. Any reference to the sex act at ALL traditionally is considered vulgarity, ESPECIALLY with animals, I would think.
To answer you, saying "I'm screwed" technically IS vulgar but has become more accepted in current society. However, you would not use the term "I'm screwed" to say that you made a mistake; rather you would say that you "screwed-up". You might actually be "screwed" because of your "screw-up", meaning you might find yourself "beset with unfortunate circumstances that seem difficult or impossible to overcome or in imminent danger" due to whatever you did that left something "Broken, damaged, inoperative or having only partial functioning". Hope this helps