I offer two idioms for your consideration: quit while [one is] behind and cut [one's] losses.
But first, let's look at the more traditional phrase "quit while [one is] ahead." Christine Ammer, The American Heritage Dictionary of Idioms, second edition (2013) has this entry for that phrase:
quit while one's ahead Don't try to improve on something that is already accomplished, as in Those drapes we hung are even enough—let's quit while we're ahead. This idiom also implies that further action runs the risk of spoiling something. Also see LEAVE WELL ENOUGH ALONE.
According to Charles Doyle, Wolfgang Mieder & Fred Shapiro, The [Yale] Dictionary of Modern Proverbs (2012), the expression "quit while you're ahead" goes back only to 1919:
Quit while you're ahead
1919 Charles E. Van Loan, Score by Innings (New York: George H. Doran) 245: "'That's right!' growls Sam Horgan, who was down on the floor with the dice. 'Quit while you're ahead, you cheap skates!'"
Within fifty years, however, people had begun occasionally using a variation on this expression that comes much closer to the sense that the posted question requires: quit while [one is] behind, meaning to stop making things worse by continuing to pursue a losing or failing course of action.
For example, from Ohio AFL-CIO, News and Views (1968), quoting a column by James Reston in the New York Times that was reprinted in the Columbus [Ohio] Citizen-Journal on November 8, 1968 [combined snippets]:
"The trouble with Hubert Humphrey is that he probably won't quit while he's behind [after losing to Richard Nixon in the 1968 presidential election], or enjoy the pleasures of defeat, He will rest for a few weeks and then get back in the political battle, ...
The expression also appears in David Profumo, The Weather in Iceland (1993) [combined snippets]:
Kit said he was getting bored with the low strike-rate, so we decided to roll the fishing grounds for just two more days before packing it in. This is usually a mistake (it's better to quit while you're behind, than compound your failures), but for once we had a real run of luck, and managed a double-header.
And from David Berg, The Trial Lawyer: What It Takes to Win (2006):
On the other hand, if your witness cratered on the stand—was so bad it's doubtful redirect can help her—quit while you're behind. Ask a few questions if you have to, but add nothing new to stir up significant recross.
The other relevant idiomatic phrase, which has much the same sense as "quit while you're behind," is cut [one's] losses. Jeffrey Moore, Prisoner in a Red-Rose Chain (1999) uses both expressions in the same sentence:
Milena was looking me in he eye. I looked away, towards my mood ring: black. Should I cut my losses, quit while I'm behind, admit I haven't the faintest idea of what I'm talking about?