Like the other two answers, "the pea under the mattress" refers to a small defect that ruins something, rather than necessarily a defect that someone is responsible for ruining their own efforts. It is a reference to the Hans Christian Anderson story "The Princess and the Pea" (originally, "Prinsessen paa Ærten").