Defined by Oxford Dictionaries as:
a person's ability to cope well with difficulties; spirit and resilience.
It is commonly seen in the phrase:
(put someone) on their mettle
(of a demanding situation) test someone's ability to face difficulties.
‘there were regular public meetings where local MPs were put on their mettle and remorselessly pilloried’
Other constructions are possible too, for example:
test their mettle, apparently an American version (thanks, @PeterCordes)
show your mettle, etc.
According to Merriam-Webster, mettle was, as @Sparhawk points out in the comments, originally simply a variant spelling of the word metal (which dates to at least the 13th century), and it was used in all of the same senses as its metallic relative. Over time, however, mettle came to be used mainly in figurative senses referring to the quality of someone's character, with a first known usage in 1581. Today, both words can mean "vigor and strength of spirit or temperament," but only metal is used of metallic substances.