To throw up the sponge is a common expression which originated in the world of boxing and is now an everyday phrase.
Boxers as you know use sponges often dipped in vinegar to wipe the sweat and blood off their face. When one of the fighters is no longer physically able to continue and wishes to give up the fight, the convention is that he or rather his trainer throws his sponge in the middle of the ring. The other fighter is then declared the winner.
So that to throw up the sponge means to give up the fight out of sheer exhaustion or disgust or to acknowledge defeat.
A common variant is "to throw in the towel" where the towel is obviously serving the same purpose as the sponge.
As for dropping the pen this is just circumstantial to the clerk's activity and I don't believe there is an actual idiomatic expression behind this part of the sentence.