In pre-Civil War America, slaves who were "acting up" would be threatened by telling them the master was going to sell them "down the river." That meant hard labor on a cotton plantation. It was a common punishment for recalcitrant slaves.
Since the main rivers in the then United States flowed generally north to south, in a progression of free states to border states to deep-Southern slave states, that represented a worsening of condition and a psychological loss of any kind of hope.
William Clark, of Lewis & Clark fame, was accompanied on the expedition by his slave York, a black man, who was promised his freedom and a portion of the rewards for his service on the more-than-two-year trip. Clark reneged, and when York got "uppity" about it, threatened to sell him down the river to a plantation.
Mark Twain's The Adventures of Huckleberry Finn describes a journey down the Mississippi River on a raft by Huck and his friend Jim, an escaped slave, who thought they were going to make the turn at the Ohio River but to their horror discovered that they had missed the turn in the night and were headed straight "down the river" where things just got worse and worse.