Is "I'm screwed" a rude expression, or can it be used when someone tries to say they made a mistake?
I overheard it from someone who seemed to have failed at his task.
It's something you wouldn't say to your grandmother, perhaps, but it is a very common euphemism people use to avoid saying something stronger. It's used in everything from pop songs to commentaries by TV talk show hosts.
Seriously, in most walks of society screwed is considered merely informal these days, not rude.
Robusto's response addresses the rudeness aspect of your question.
With regard to the second part, whether I'm screwed is "used when someone tries to say they made a mistake": I think you're confusing I'm screwed (which as the comments tell you means approximately "Aw, jeez, I'm in trouble") with I screwed up, which does mean "I've made a mistake".
Another version of screwed up, made famous in The Right Stuff, is screwed the pooch; this is somewhat intensive, but the conventional intensifier is royal: "I screwed up royally."
Both expressions (and the "stronger" version to which Robusto alludes) may be used in second or third person as well as first, in either the singular or plural, and in any tense, aspect or mood.
Screw up is also frequently used adjectivally ("That's a really screwed-up piece of design") and nominally ("I'm getting tired of Carol's screw-ups"). The nominal version of screw in this sense is more often the gerund screwing than the infinitive screw (which tends to be reserved for the sexual sense); I cannot recall encountering an adjectival use.