'Dukes' as fists in early U.S. sources
James Maitland, The American Slang Dictionary (1891) has this entry for "Dukes or Dooks":
Dukes or Dooks (P. R.), the hands or fists. "Put up your dukes" is an invitation to fight or spar.
The abbreviation "P. R." stands for "Prize Ring," referring the ring where boxing matches occur.
The earliest slang dictionary reference to dukes as "hands" that I've been able to confirm, however, is considerably older. From George Matsell, Vocabulum: Or, The Rogue's Lexicon (New York, 1859):
DUKES. The hands.
Matsell is identified on the title page of this book as "Special Justice, Chief of Police, Etc., Etc." and his publishing company (George W. Matsell & Co) as "Proprietors of the National Police Gazette." Henry cites this source in his answer, but he doesn't mention that the dictionary was published in the United States by a New Yorker who claimed to be a police official.
A very early instance of duke in the sense of fist an in the context of fighting appears in Samuel Chamberlain, My Confession: Recollections of a Rogue (1855–1861[?]) [combined snippets]:
One Sunday on my way to Church I was insulted by a rough, and on my remonstrating with him for using profane language on the Holy Sabbath, he with a fearful oath struck at me. Now while I was ready to forgive the sinner for his insult to me, I felt it was my Christian duty to punish him for his blasphemy. With my right I neatly stopped his blow, and landed a stinger on his "potatoe trap" with my left "duke," drawing the "Claret" and "sending him to grass." The Rowdy got up and ran down Chardon Street, and I turned the cross over, when I saw one of our good Deacons with his two lovely daughters passing. From their looks I knew they had witnessed the little unpleasantness. This alarmed me at
first, but when I caught sight of a merry twinkle in the good Deacon's eyes and an admiring glance from the young Ladies, I felt safe.
Chamberlain was born in New Hampshire and lived in New England for much of his life, aside from adventures in Texas and elsewhere during the Mexican–American War and the Civil War. Unfortunately, his memoir is of uncertain age; J.E. Lighter, Random House Historical Dictionary of American Slang (1993) reports it as circa 1859, but another source reports that Chamberlain probably wrote it over a period of seven years between 1855 and 1861.
A more precisely dated occurrence appears in "Determined Fight: Between Nolan and Hicks for One Hundred and Twenty Sovereigns," in the New York Clipper (March 3, 1860):
[Round] 31. Joe planted a severe rib bender on the left side of Hicks' chest, and the blood trickled directly. Hicks returned with his left, but it was not high enough. Jack received another right-handed one on the ribs, but played well with the left duke, until Nolan's nose bled afresh. The character of the rounds, up to the forty-fifth round, were of the same even character—Jack proving as good as his master.
More than a dozen additional unique instances of "left duke" or "right duke" or appear in the Clipper and other newspaper sporting sections over the remainder of the year 1860, providing extremely strong evidence that the usage was well established in New York boxing patois by the end of that year.
A somewhat later instance of a left duke appears in Charles Henry Webb, John Paul's Book: Moral and Instructive: Consisting of Travels, Tales, Poetry, and like Fabrications (1874), which very conveniently lays out a wide range of then-current slang and metaphor from the ring:
I measured my man in a moment, applied my mental callipers to his muscle, saw at a glance that he beat me so far as biceps went, took into consideration the fact that he was much older than myself as well, and concluded to forgive him. I am glad that I did; it is always Christian-like to pass little remarks of this kind over without notice unless you are morally sure that you can whip your man.
Nor do I know exactly that I would care to tap the claret, smash the smeller, upset the snuff-tray, damage the optics, close the peepers, devastate the oglers, smite the conk, counter on the kisser, spoil the potato trap, mash the mug, and generally macerate the mouth of Milton [Sandorf], to say nothing of demolishing his bread-basket and laying waste and capsizing his apple-cart ; for, though capable of reaching out with my right bower and putting in my Left Duke—my terrible Left—in an appalling style, I have nothing in particular against Mr. Sandorf.
Webb was born in New York and lived at various times in New York, California, and Massachusetts.
The Random House Historical Dictionary of American Slang's entry for duke notes an instance from the November 10, 1866, issue of The National Police Gazette (Matsell's tabloid) that reads in truncated form as follows:
The watch and chain...are in the "dukes" of...Robert White
And from "The Bolster and Warburton Fight: A Very Tame and Uninteresting Affair: Siz Rounds Fought in Ten Minutes: Nobody Hurt—Everybody Dissatisfied," in the New York Clipper (February 33, 1867):
Warburton is an Englishman, born in Sheffield, Eng., about 1834, is 5ft 8in., and was 11st 6lb (160lb) on the day of his "maiden essay." He was backed from Hartford, where he once had numerous friends and could at one time make a good display with the gloves, although he seems to have entirely forgotten the use of his dukes, by his present engagement. He is said to have figured in the P. R. "at 'ome," but we never could learn who with.
'Put up [one's] dukes' in early U.S. sources
Although, various answerers have suggested that the phrase "put up [one's] dukes" may go back to 1859, the two earliest instances I have found of it in print are from 1865 and 1866 in the New York Clipper, a newspaper that focused on boxing and other sports. First, from "Almost a Fight—Old Slumbo vs. The Woonsocket Boy," in the New York Clipper (March 11, 1865):
The spectators at the Marley and Devine mill in California came within an ace of witnessing another treat while returning to San Francisco on the steamboat. Brock, who rejoices in the comical cognomen of Old Slumbo, had been taken a taken a snootful or two of Jersey cider, and felt his oats—he therefore called on Barney Farley, better known as the Woonsocket Boy, to put up his dukes, as he thought he could knock all the fight out of him in about three minutes. Slumbo was stripped to the pelt, eager for a clinch, and Barney was rapidly divesting himself of all superfluous toggery, in order to rush to his embrace, when the captain of the steamer, thinking there had been glory enough in one day, put in an appearance and nipped it in the bud.
And from "The American P. R. Championship: The Californian Heard From: He Is Ready to Come to Terms with Woods," in the New York Clipper (February 3, 1866):
We have all along been too thoroughly convinced of the integrity and love of fair dealing characterizing Mr. [William] Davis [the Californian], to allow ourselves to lend an ear to such idle and malicious assertions [as that Davis had challenged Woods to a fight to gain notoriety but without intending to enter the ring against him], and we are gratified to have it in our power to refute the same by publishing the following business-like document from William of Santa Fe; which proves that he has not gone to California by steamer, and that he is ready to put up his dukes with whoever has the courage to pick up the gauntlet he has thrown down.
'Dukes' as fists and 'put up [one's] dukes' in early British sources
The earliest match that the British Newspaper Archive finds for dukes in the sense of "fists" is from "Prize Ring in America," in Bell's Life in London and Sporting Chronicle (January 11, 1868) [combined OCR snippets]:
... audience. The scene in which the Chelsea Snob and the Hebrew appear is truly ludicrous. Phil having lost the use of his dukes seizes the Snob with his teeth, .wÂ«rtn* -ill have uund meat, thus down roars of laughter, treat which s[h]ould not missed by l[o]vers of pleasure ...
The illegible form of the latter part of this quotation is owing to the fact that I have access only to the OCR rendition of the article snippets in question (because I don't have a subscription at the BNA site). But the cited article appears to describe a scripted stage show in the United States involving a fight between "the Chelsea Snob" and "the Hebrew." It is striking that the first instance of dukes as fists to appear in a London sporting paper involves an article reporting from the United States.
The earliest instance of "put up [one's] dukes" that the British Newspaper Archive finds is from "Violent Assault in the City," in the Clerkenwell News (September 13, 1869) [combined OCR snippets]:
Shortly afterwards the prisoner overtook him Creechurch-lan[e], and asked him what h[e] had left him for, and told him to put up his dukes. Prisoner then struck him several times in the face and scratched it, and ultimately knocked him down, and struck him on the head with his boot, cutting it open end causing him to bleed profusely.
The timing of the U.S. and British recorded instances that I've cited casts doubt on the presumption that "put up your dukes" originated in England. The phrase does not seem to go back much farther than the middle 1860s, nor does dukes as fists leave much of a paper trail before 1859. For instance, John Bee [Badcock], Sportsman's Slang: A New Dictionary of Terms Used in the Affairs of the Turf, the Ring, the Chase, and the Cock-Pit (1825) makes no mention of dukes (or dooks) as fists, despite Badcock's very particular interest in slang associated with pugilism.
The "Modern Flash Dictionary" appended to Sinks of London Laid Open: A Pocket Companion for the Uninitiated (1848), has this seemingly unrelated entry but nothing else for duke, dukes, or dooks:
Duke of limbs, a deformed person
John Hotten, A Dictionary of Modern Slang, Cant, and Vulgar Words: Used at the Present (1859) has entries for dookin, duke, and Duke of York, but none of them seem on point:
DOOKIN, fortune telling.
DUKE, gin. Ho[usehold] Words, No. 183.
DUKE OF YORK, take a walk.
Ducange Anglicus, The Vulgar Tongue: Comprising Two Glossaries of Slang, Cant, and Flash Words and Phrases, Principally Used in London at the Present Day (1857), notes dookin (in the sense of fortune-telling) and cites W.A. Miles, "Poverty, Mendacity, and Crime, or the Facts, Examinations, &c. upon which the Report was founded..." (1839) as the source of that entry, but doesn't mention dukes or dooks. The second edition of Anglicus's book, The Vulgar Tongue: a Glossary of Slang, Cant, and Flash Words and Phrases Used in London from 1839 to 1859, second edition (1859) likewise provides no information on either term.
The first British slang dictionary to note dukes (or dooks) in the relevant sense is John Hotten, The Slang Dictionary: Etymological, Historical, and Anecdotal (1874), which also mentions the phrase "put up your dukes":
Dukes, or DOOKS, the hands, originally modification of the rhyming slang, "Duke of Yorks," forks=fingers, hands—a long way round, but quite true. The word is in very common use among low folk. "Put up your dooks" is a kind invitation to fight.
This entry is quite interesting because it is the first dictionary effort I'm aware of that tries to explain where the expression came from. Moreover, Hotten was a serious researcher and was writing at a time not distant from the earliest recorded instances of "put up your dukes." But despite these considerations, a few points make me less inclined to accept Hotten's explanation than I might otherwise be:
As late as the 1870 edition of his dictionary, Hotten did not have any entry for dukes/dooks.
As late as the 1864 edition of his dictionary, Hotten was reporting that in rhyming slang "Duke of York" meant "take a walk."
Although British instances of "put up [one's] dukes" appear at least as early as 1869, U.S. instances of the phrase appear at least as early as 1865.
The earliest (that I've been able to confirm) instance of dukes as fists is from 1859 in the United States—15 years before the 1874 edition of Hotten's book asserted that term originated in British rhyming slang.
So if Hotten is correct about the origin of the term from rhyming slang, either he must have misunderstood the meaning of the rhyming slang "Duke of York[s]" in the period between 1859 and 1864 or that meaning must have changed at some point from "take a walk" to "fingers"—a rather odd jump. Moreover, during the same period, the slang usage of dukes to mean "hands"—which was already to be found in the New York demimonde in 1859— must have evolved (either in Britain or the United States) to produce the phrase "put up your dukes," which caught on among boxing aficionados in the U.S. by 1865 and in Britain by 1869; and then, by 1874, both dukes and "put up your dukes" must have become sufficiently popular in Britain for Hotten to have taken notice of it. This isn't an impossible scenario, by any means, but it is rather more iffy than Hotten's brisk 1874 summary might lead one to believe.
Albert Barrère & Charles Leland, A Dictionary of Slang, Jargon & Cant, volume 1 (1889) doesn't seem to buy the Hotten theory of the origin of "put up your dukes":
Duke of York (rhyming slang), walk or talk.
Dukes or dooks (popular and thieves), the hands; from the gypsy dūk, dook, which refers to palmistry; "it is in his dook," meaning "it is in his fate," became "it is in his hand." [Cited example:] Then he began to push me about, so I said I would not go at all if he put his dukes (hands) on me. Then he rammed my nut (head) against a wall and shook the very life out of me.—Horsley: Jottings from Jail.
J.W. Horsley was chaplain at Clerkenwell Prison in England, but his book Jottings from Jail: Notes and Papers on Prison Matters was published in 1887, so it is difficult to say how far back the slang usage of dukes went in this narrated instance. It is also interesting that Barrère & Leland reports two meanings of "Duke of York"—"walk or talk"—neither of which has a close connection to fingers.
On the other hand, Barrère & Leland's claim that dukes derives from a Romany term suffers from the fact that the earliest instances of that usage, during the period 1859–1868, are from the United States (where, in the mid-nineteenth century, Romany presence was negligible), not England (where it was far more significant at that date). According to the Wikipedia article on Romani people,
Romani began emigrating to North America in colonial times, with small groups recorded in Virginia and French Louisiana. Larger-scale Roma emigration to the United States began in the 1860s, with groups of Romanichal from Great Britain. The largest number immigrated in the early 1900s, mainly from the Vlax group of Kalderash.
But since the term dukes as fists was already recorded in the United States by 1859, it predates the earliest major wave of Romani migration to the United States.
Here is the chronology of instances related to dukes as fists and the phrase "put up [one's] dukes" that I found in the course of my research:
1859 (U.S.) Matsell, Vocabulum includes the entry, "DUKES. The hands."
1855–1861[?] (U.S.) Chamberlain, My Confession, refers to throwing a punch with "my left 'duke'."
March 3, 1860 (U.S.) An account in the New York Clipper of a prize fight uses the term "left duke" and is followed over the next 10 months with more than a dozen other news articles—in the Clipper and elsewhere—that use the term "left duke" or "right duke."
March 11, 1865 (U.S.) A newspaper article uses the phrase "put up his dukes."
February 3, 1866 (U.S.) A second newspaper article uses the phrase "put up his dukes."
November 10, 1866 (U.S.) Matsell's National Police Gazette refers to "dukes" holding a watch and chain.
February 23, 1867 (U.S.) A newspaper article criticizes a boxer for having "entirely forgotten the use of his dukes."
January 11, 1868 (Britain) A newspaper story called "Prize Ring in America" refers to a character in a stage skit who has "lost the use of his dukes."
September 13, 1869 (Britain) A newspaper article reports an assault in London in which the assailant challenges the victim to "put up his dukes."
1874 Hotten, A Dictionary of Modern Slang... includes an entry for dukes that also notes the phrase "put up your dooks" and claims that the term arose from rhyming slang.
This chronology offers fairly strong circumstantial evidence for the hypothesis that both dukes as fists and "put up [one's] dukes" as a set phrase originated in the United States and then jumped the Atlantic to catch on in Britain. Of course, a single relevant instance of dukes from Britain prior to 1859 would puncture this hypothesis. But at the very least, the instances I have noted here encourage a healthy skepticism regarding the presumption that the phrase "put up your dukes" originated in Britain.