The precursor phrase, 'burn in [one's] pocket'
Th antecedent to "burn a hole in [one's] pocket" was almost certainly "burn in [one's] pocket," as user66974's very useful answer points out. The phrase goes back considerably farther than the Phrase Finder quotation in that answer suggests. Phrase Finder cites the OED's instances of "your letter, which burnt in my pocket" (from 1740) and "it [money] burns in their pockets" (from 1768).
A Google Books search, however, finds examples going back to the late seventeenth century.
From the preface to A Warning-Piece to All Drunkards and Health-Drinkers (1682):
The bewitching, besotting nature of Drunkenness : It doth not turn men into Beasts, as some think, for a Beast scorns it : I do n't know that ever I saw a Beast drunk (unless it were a swine) in my life. But it turns them into Fools and Sots, dehominates them, turns them out of their own Essences for the time, and so disfigures them, that God saith, Non est haec Imago mea, This is not my Image ; and so cares not what Judgments he lets fall upon their heads : And this Infatuation is more eminently seen in the Poorer sort that earn their Money hardest, and pay most for their Drink : for when others pay their Money, these pay their Time also, which is more than their Money, besides the loss of Trade and other possible advantages : That others drink Sack cheaper than they pay for their Beer and Ale, all things considered ; and their poor Wives and Children by this Means are in want of Bread for their Mouths : And will not God be avenged upon such a sin as This? Their Money burns in their Pockets, but it will burn worse in their Conscie[n]ces, if ever God shew them Mercy.
From John Phillips, A Pleasant Conference upon the Observator and Heraclitus Together with a Brief Relation of the Present Posture of the French Affairs (1682, page 23):
Belfagor. The same Relation informed me likewise, that as he was a great Racer himself, so he thought it also convenient to keep a running Nag: To which purpose when the carter brought the Money to pay his Father for the surrender of the Garrison of Iusticia, the young squire knocks off the hoops of one of the Firkins, cram'd his pockets, and presently tript it away to the chief City of Plotters land; thither being come, his Money burning in his breeches, he repaired for a Cooler to a Reverend old Matron, whom your Highness well knows, as having been long famous for sin and iniquity, called Betty Buly.
From Peter Pett, The Happy Future State of England (1688):
As hidebound as king Iames found Parliaments afterward (for as I said) he in his Speech in Parliament Anno 1620 mentioned, That in all his Reign he had but 4 Subsidies and six Fifteenths, yet their belief of that Popish Gun-powder Plot fired the Zeal of their Supplyes, and (as i may say too) made their Money burn in their Pockets, and pass with speed into the exchequer, and with a Salvo to the Caution about not drawing that Act into a President, &c.
And from George Farquhar, The Inconstant, or the Way to Win Him (1702):
Duretete. My friend has forsaken me, I have abandoned mv mistress, my time lies heavy upon my hands, and my money burns in my pocket. But, now I think on't, my Myrmidons are upon duty to-night; I'll fairly stroll down to the guard, and nod away the night with my honest lieutenant, over a flask of wine, a story, and a pipe of tobacco.
From Jonathan Swift, "Bon Mots De Stella" (by 1744) in The Works of D. Jonathon Swift, volume 8 (1746):
Dr. 5heridan, who squandered more than he could afford, took out his purse as he sat by the fire, and found it was very hot ; she [Stella] said, the reason was, that his money burnt in his pocket.
Stella Johnson died in 1728, so the her jest about the burnt pocket occurred considerably before Swift wrote this recollection.
And (as cited in user66974's answer) from Abraham Tucker, The Light of Nature Pursued (1768):
Every body will acknowledge that the value of money arises solely from the use of it : if we had not found it commanding the pleasures and conveniences of life we should never have thought it worth our regard. Nature gave us no such desire, but we are forced to take pains in teaching children to be carefull, and those with whom such pains have proved unsuccessfull cannot rest till they get rid of their money, or as we say, it burns in their pockets.
According to J. Goujon, A Dictionary of the Idioms of the French and English Languages (1812), the expression as idiomatic in English by Goujon's time:
[French idiom:] Il est bourreau d'argent. [or in another section of the same dictionary, "C'est un tonneau percé."]
[English idiom:] He is a spendthrift—Money burns in his pocket.
We thus have instances of money burning in one's pocket (or breeches) from 1682, 1682 (again), 1688, 1702, 1746 (referring to a bon mot spoken in 1728 or earlier), and finally 1768.
The modern phrase, 'burn a hole in [one's] pocket'
From J. Fisher Murray, "The Physiology of London Life," in Bentley's Miscellany, volume 16 (1844) treats the expression as well established:
Ready money has been long known to possess this latent or concealed heat in great quantity ; long before the time of Black [a scientist] it was ascertained that money burned a hole in many a man's pocket, as well as in mine. In a paper read before the Acidulous Association, it was attempted to be proved that Black was indebted for his discovery [of latent heat] to the knowledge of this popular fact, which he applied to chemical science. Be that as it may, I never had a penny in my pocket that did not in the course of ten minutes get so intolerably hot that I was obliged to rush into a tailor's, tobacconists, picture-dealers, book-shop, or tavern, according as one or other happened to be at hand, and empty my pockets on the counter. You would hardly believe how much I saved this way in pocket linings.
Various searches turn up a number of earlier examples.
From a letter dated December 21, 1820, to the editor of The Literary Gazette and Journal of the Belles Lettres (January 6, 1821):
A man who reads or meditates a good deal upon one subject is fond of talking about it, and is apt to think, not only that the subject of his studies must be quite as pleasant and profitable to others as it was to himself ; but also, that he has the talent of imparting to them some portion at least of that enthusiasm which he has felt himself. In short, zeal in any cause, however trifling and unimportant, is like a spendthrift's money, it burns a hole in his pocket; with the advantage, that instead of impoverishing him, it adds a rich harvest of pleasure, and perhaps of renown, to the rest of his store.
From Mungo, Coultershoggle, Goslington Shadow: A Romance of the Nineteenth Century, volume 2 (1825):
"As times are in England," said the Laird, "I wad na tak land at a gift. I ne'er was mair mista'en a' my days than since I came up to England—the soil and the climate seemed to be sae muckle better than in Scotland—and the farmers seemed to be a' better aff as we cam alang the road—and then, whan we came to Lunnen, it surpassed every thing. I thought England was a fine place, and I was fear't that Sir Belfry would sell his estate before I could see it, but now, as times are, nae England for me ; my siller's [silver is] no burning a hole in my pouch yet."
From an 1829 translation of Memoirs of Vidocq, Principal Agent of the French Police, Until 1817 (1829):
"That's right, my worthy old gentleman, let your purse-strings crack ; it will be all the same fifty years hence, and you will be neither the richer nor the poorer for it : now then, my good man, throw in a few of those louis which have been burning a hole in your pocket so long.—Pray remember the cap," continued she, shaking it in the face of an old lady who seemed anxious to escape her eye.
From "Sorting My Letters and Papers," in Blackwood's Edinburgh Magazine (November 1829):
Yes! I knew “the poetical shoemaker” well; but at the time I speak of, he kept a little bookseller's shop, and thither I used to go, many a time and oft, in the days of my pocket-money, not so much to avert the predicted mishap of having a hole burned in my pocket by the newly deposited shilling or half-crown, as to pay it with delight for "another Number" of Milton, or Dryden, or Gray, or Thomson, published in a neat pocket edition by "C. Cooke, Paternoster Row."
From a letter to the editor of the Sydney [New South Wales] Gazette (May 5, 1829):
Considerable sums of sterling money are now sent up to the interior ; chests, whose contents are dispersed in various channels, and the receiver will calculate his purchases and payments in the same coin and manner in which he is paid his own salary. A man will also find it a beneficial act on the score of economy. He can now receive his allowance close to his own door, and thus, before the coin has time to burn a hole in his pocket, be passes it over to his baker, butcher, brewer, brewer, blacksmith, grocer, taylor, &c, &c. &c. ad infinitum.
From "The Old Maid and the Gun," in Blackwood's Edinburgh Magazine (February 1831):
"Oh, Johnnie," said I, "what a great chield ye have grown! the breeks that I mended for ye 'll be o' nae use to ye now, and the sarks 'll be perfectly thrown awa." He began to laugh, when I said this, wi' the same wild laugh he used to do at hame; and said, "What! auntie, always thinking about the pence yet?"—"It'll maybe be the better for you some day, if I do; for, if ye're no greatly changed, a bawbee aye burned a hole in your pocket unco soon."
From "Banks for Savings," in Andrew Thomson, A Collection, in Prose and Verse, for the Use of Schools (1835):
Within these limits, a contributor transacts with the bank just as he pleases. No one asks him questions about either his receipts or his payments ; he is the sole master of his money. All this does him good in various ways. In the first place, his money will not burn a hole in his pocket, and will neither be spent on gingerbread, if he be a little boy, nor on whisky, if he be a man, when it is lodged in the coffer of the bank.
From "Generosity of Harry Coil and Tim Taffrail," in The British Log Book; or Yarns of the Ocean (1840):
The sailors inquired into the events which had driven her [a poor women whose household goods were on the verge of being confiscated] to this distress ; and she informed them that her husband had formerly been a mate on board a merchantman, but had met with an accident which disabled him from going to sea for several months : and the expense and loss occasioned by this visitation, had so embarras[s]ed their affairs, that they became in debt, and a merciless creditor had, on the preceding day, taken her husband into execution for five pounds—Harry Coil felt the money burning a hole in his pocket.
From "Canada and Corn," in The Spectator (November 19, 1842):
"We call this a most brainless burlesque on Liberal colonial policy. Liberal it is with a vengeance—the liberality of one whose money, in the vulgar phrase, burns a hole in his pocket. Who but must recognize the Colonial Secretary who, in 1831, was a party to that famous policy which consigned the civil list to French Canadian discretion! Who but recognize the Minister of the twenty million gift to the Planters!
From Mr. M'Combie, "Australian Sketches: The Settler" in Tait's Edinburgh Magazine (1843):
Experience might have taught him wisdom ; and it might now have been expected that he would not have spent all. Never, however, was there a more complete mistake than to suppose an outlandish settler would keep money,—it would burn a hole in his pocket : so it all goes, and, in the end, he is as poor as ever.
All of these early examples are from England, Scotland, and Australia. The expression shows up in the United States by the late 1860s.
It seems extremely likely that the phrase "burn a hole in [one's] pocket," in reference to money, evolved from the earlier "burn in [one's] pocket," which likewise referred to money—and to how difficult and uncomfortable it is (metaphorically speaking) to keep money in your pocket when you have the urge to spend it.
Although the OED cites occurrences of the older phrase from 1740 and 1768, it was actually in use by 1682. And although the OED cites an example of the newer phrase from 1857, I found examples of it from as early as 1821. Both English expressions seem to have originated in Great Britain.