Anglo-Norman (the French spoken in England) already had both meanings, and so English inherited them.
The law dictionary Termes de la Ley (originally printed and authored by John Rastell in 1527; new editions published in twenty-nine editions up to 1819 (Tarlton Law Library)) features this use of recoup in the 1685 edition, as archived on Early English Books Online:
AGent & Patient is, when a man is the doer of a thing and the party to whom it is done; as where a Woman en∣dows her self of the fairest pos∣session of her husband. So if a man hath ten pounds issuing out of certain land, and he disseises the Tenant of the Land in an Assise brought by the Disseisee, the Disseisor shall recoup the Rent in the damages; so that where the mean profits of the land in such case were to the va∣lue of 13 l. the Disseisee shall re∣cover but three pounds. (p. 33)
In other words, the person seizing land (the Disseisor) shall retain the rent owed by the Tenant while also paying the previous land owner (the Disseisee) the remaining sum of money. Recoup already means this here (meaning 2 in the OED).
Meanwhile, a usage of French recoup in the text means much the same thing:
car lou le Grantor doit receive xx.li. damages, & pay x.li Rent, il puit aver receive forsq̄ le xli. solem̄t p̄ les damages, & le Grantee puit aver recoup. & retaine arere le aut' x.l. ē ses maines {per} voy de deteiner pur son Rent, & issint {per} ycel poet aver save son Action. (p. 144)
because the Grantor must receive 20 pounds damages, and pay 10 pounds rent, he has the power to have received only the 10 pounds alone for damages, and the Grantee can have recouped and retained back again the other 10 pounds out of these left for the purpose of withholding on account of his rent, and thus like that equally this one has the power to have saved his law suit. (Translation my own; apologies for errors)
The grantee recoups and retains the 10 pounds that he otherwise would've been paid for rent. So at least in Termes de la Ley, the earlier meaning of recoup is preserved.
When did the meaning shift to meaning recompense? I focus on Termes de la Ley because it shows that English and French usages of recoup are strongly connected. A similar thing is true for the recompense meaning of recoup. Recoup could have meant recompense in French as early as the 15th century. The Anglo-Norman Dictionary places this meaning of recoup ("to recompense") in the 15th century:
[finan.] to recompense: demaundant dower quar eux ne purrount estre recoupés en l'accion d'acompt Readings 195. (Readings and Moots at the Inns of Court, ed. S. E. Thorne, Selden Soc. 71 (1952). date: c.1420-1489)
So English legal clerks, writing in both Anglo-Norman and English, would have been familiar with the recompense meaning already. So when legal translations of older law, written in French, are translated or recast into English in the 17th century, both meanings of recoup coexist for a while. The more straightforward meaning etymologically comes through first, but the second one shows up within a few decades. They coexist for a few centuries.
So what we're seeing is not a shift from one meaning of recoup to another, at least in English. What we're seeing instead are two distinct French meanings translated into English (and then evolving from there) as the Anglo-Norman law code shifted into the English language.
