The Oxford English Dictionary has this definition under "serve, v.1.," along with the earliest cited usage and a directly comparable usage:
- to serve out: to punish, take revenge on; to retaliate on (one) for something objectionable. colloq. (orig. Boxing slang). Also (Hunting slang), to ‘punish’ or smash (a fence).
1817 Sporting Mag. 50 18 The butcher was so completely served out, that he resigned all pretensions to victory.
1863 C. Kingsley Water-babies v. 183 I will serve you out for telling the salmon where I was!
I see no evidence that it refers to killing an animal. The same phrasal verb is used to refer to giving food or drink, with an earlier first cited usage:
- c. to serve out, to distribute or deal out (food, ammunition, etc.) in portions. [...]
1793 J. MacDonell Diary 15 Aug. in C. M. Gates Five Fur Traders (1933) 101 Our Bourgeois came up with us and ordered each man a dram, which I served out to them.
It may be that this form of service was then generalized to other subjects or objects. In the 1824 collection of anecdotes Boxiana, these uses appear that emphasize what is being served out or dealt:
In closing, the punishment which Carter served out to his opponent was tremendous in the extreme [...] (p. 316)
Much punishment was served out on both sides [...] (p. 349)
Even without punishment being specified, then, the boxing context would convey what is meant in the following metaphor of boxer as a customer of blows:
However, at all events, Ned (the boxer) is a customer not easily to be served out (p. 108)