From Hugh Rawson, A Dictionary of Euphemisms and Other Doubletalk (1981):
Jiminy Cricket. The cute Walt Disney character notwithstanding, this is a euphemism for "Jesus Christ," on a par with Judas Christopher, Judas Priest, cripes, and jingo. The "Jiminy" comes from "Gemini," which goes back to at least 1664, and which may derive from the Latin Jesu domine. "Jiminy" is sometimes used alone, as in "By Jiminy" or—perhaps a transitional form—"'Oh, geeminy, it's him,' exclaimed both boys in a breath" (Mark Twain, The Adventures of Tom Sawyer, 1876). "Jiminy" also may be finished off in other ways besides the cute "Cricket," e.g., Jiminy Christmas, Jiminy crackers, Jiminy criminy, Jiminy cripes, and Jiminy whiz.
John Bartlett, Dictionary of Americanisms (1848) has this entry for the word, which does not indicate any connection between Gemini and Jesu domine:
JIMINY. By Jiminy! An exclamation. Originally, gemini, or the Castor and Pollux of ancient mythology; names by which the old Romans used to swear.
A search of Early English Books Online returns one early (but quite ambiguous) potentially relevant match for "by Gemini," from Holinshed's Chronicles (1587):
More welcome than Terpsicore was to the towne of Troie. / Sea-faring men by Gemini conceiue not halfe my ioie. / Strong Hercules to Theseus was neuer such delight, / Nor Nisus to Eurialus as I haue in this sight. / Penelope did neuer thirst Ulysses more to see, / Than I poore Norwich hungred haue to gaine the sight of thee.
It seems more likely that the sea-faring men are feeling joy as they use the constellation Gemini to steer by than that they are feeling joy by a euphemism along the lines of "jumping Jehosephat." I haven't yet found the 1664 source that Rawson alludes to.
The earliest use of jiminy as an expostulation that I've found in various searches is from Woman's Will: A Comedy, in The New British Theatre: A Selection of Original Dramas Not Yet Acted, volume 4 (1815):
Old H[arcourt]. I cannot think of it.—For as he is unquestionably my lawful heir——
Lucy. Lawful heir! jiminy, jiminy, how you provoke me! Shall a trifle like this be set in opposition to the force of love? Omnia vincit amor, as the poet says ; and which in English means—that is, as Mr. William, (he was bred at Oxford) informs me—“Love subdues pretty girls,” and this, indeed, he kindly taught me long ago.
On the other hand, the euphemistic use of Jiminy for Jesus seems quite strong in Henry Paul, Dashes of American Humour (1852):
"Jiminy Cranks! Yew ain't agoin to ride in that thing, are yeow? It looks like a patent coffin!" said our Yankee friend, gazing with an air of curious interest at the cab, and crossing to examine the position of the perch. "What a pesky quare go for the driver to set behind! There's where these things and cabs arn't alike."
Update (September 27, 2019): Early occurrences of variant spellings of 'Jiminy'
Prompted by the suggestions in Green's Dictionary of Slang (cited in user067531's very useful answer), I searched for some variant spellings of jiminy and found these early instances:
From William Wycherley, The Country-wife: a Comedy Acted at the Theatre Royal (1675):
Hor[ner]. I thought so, for he is very like her I saw you at the Play with, whom I told you, I was in love with.
Mrs. Pin[chwife]. O Jeminy! is this he that was in love with me, I am glad on't I vow, for he's a curious fine Gentleman, and I love him already too.
From Aphra Behn, Sir Patient Fancy: A Comedy: As It Is Acted at the Duke's Theatre (1678):
Sir Pat[ient]. Hah, so young a Bawd! —tell me Minion, —private meeting! tell me truth I charge ye, when? where? how? and how often? oh she's debauch't! —her reputation's ruin'd, and she'le need a double Portion. Come tell me truth, for this little Finger here has told me all.
Fan[y]. Oh Geminy Sir, then that little Finger's the hougesest great Lyer as ever was.
From a 1684 translation of The Idylliums of Theocritus with Rapin's Discourse of Pastorals:
Here Betty take the Boy, and stay at home, / Call Pretty in, and wait here till I come. / O Jemminy, dear Gorgo, here's a throng, / I wonder how we two shall get along: ...
From (as noted in Green's Dictionary of Slang), Thomas D'Urfey, The Comical History of Don Quixote as it is acted at the Queens Theatre in Dorset-Garden (1694–1696):
Teresa takes the Letter.
Teres[a]. Ah Gimminy, I could eat the Letter up methinks: —well dear Sancho, or dear Governor, here I am come to thee at last; good Lord Mary! I can but think upon his former words, which odsdiggers I could n'er have believe then, tho now I find 'em true ...
And from Edward Ravenscroft, The Canterbury guests, or, A bargain broken a comedy : acted at the Theatre-Royal (1695):
[Mr. Justice] Greed[y]. But what do I prating here, when Dinners going up.
Dash. Gemminy Sir, how you are garnish'd out, as if you were to be serv'd up for a standing Dish — more for ornament, than use —let me help you off with your Cloak. Some body has pin'd a dish clout to your back.
From these early matches, it seems possible that the variants jeminy, geminy, jemminy, gimminy, and gemminy all may antedate the earliest print occurrence of the current standard form jiminy. It is also striking that the expression appears so frequently in plays of the period 1675–1695, often in lines spoken by servants or other people of common background. Evidently, the playwrights were familiar with the spoken expression but differed on how to spell it.