Leo Rosten, Hooray for Yiddish (1982) has a lengthy entry about this phenomenon under the modest title "sh—":
sh—
shm—
Yinglish. As in Yiddish, these prefatory particles mock or negate the word they prefix.
- Sh— designates scorn or dismissal when used to prefix a word" "Sick-shtick, he should be in the office!" ...
In addition to such snickery, the sh— sound introduces an astonishing number of Yiddish/Yinglish words to describe character, and freights the with disdain: shlemazl, shlemiel, shlepper, shlock, shlump, shmatte, shmeer, shmegegge, shmendick, shmo, shmontses, shmuck, shnorer, shnuk, shreck, shtunk, shtus, shvantz
...
- Shm— extends the scorn (as seen in the list above) by adding the nasal phoneme m to the soothing but ironic sibilants. "Clone-shmone, how will they keep their identity?" ...
The fusion of the derisive shm— to a word in order to deflate it has been called "mock-language" (by Noah Prylucki). Max Weinreich remarks that the shm— prefix "became universalized only toward the end of the nineteenth century; earlier, forms with shp—, shm—, or simply with p- or m- would do" (History of the Yiddish Language, page 623).
Examples of the shm— gambit have been found by Ernest Henri Levi in German dialects as far back as the thirteenth century. The oldest written evidence (so far) of a shm— usage is in a manuscript of 1600, written near Augsburg.
Some of the mock-words Jews used were part of "secret" language—the disguising of certain words about whose usage Jews harbored anxiety (e.g., the Trinity).
in German the use of this mock-mechanism (sh—, shm—) was never extensive, and has progressively declined since the Middle Ages; the cases recorded from modern German dialects seem to be loans from Yiddish. This, of course, does not preclude the beginning of the phenomenon from having been German, but we have no complete certainty....
—MAX WEINREICH, op. cit., pages 623–24
Although Rosten is more of a popular writer than a scholar, Max Weinreich is a pure academic. His massive History of the Yiddish Language was first published (in Yiddish) in 1974, five years after his death. It appeared in English translation in 2008 in an edition published by Yale University Press that runs to almost 1,800 pages.
I take seriously Rosten's basic argument that the shm— mock repetition mechanism goes back a long way in Yiddish and German, and that (as Weinreich says) "it became universalized only toward the end of the nineteenth century."
I searched for a number of mock-repetition words in Google Books, and the earliest (and most popular) one was fancy-shmancy (or fancy-schmancy), which appears in publications dating back to the 1940s. For example, from Stephen Longstreet, The Last Man Comes Home: American Travel Journals, 1941-1942 (1942) [combined snippets]:
The pushers are stiff with shoving it along all night. The warehouses are full. Crates of chickens speak to each other, compare eggs. Golden-brown corn, Country Gentleman, sweet on the cob, great tubs of salt butter, the good bitter tang of dill pickles, the tubs of schmaltz herring, the fancy-schmancy pheasant, rabbit and deer cadavers for the tender, trained stomachs of ...
From Carlton Brown, Brainstorm (1944) [combined snippets]:
The next day I told Helen about the perplexing state of affairs I had run into at Max's, and said that I guessed he was getting a little too fancy-shmancy for me, tasting a suspicion of envy as I said it.
From American Aviation, volume 10 (1946) [text not visible in snippet view]:
LITTLE KNOWN FACTS Here are 3 new Little Known Facts About Well-Known Planes, each of which has gotten its sender a fancy-shmancy commission as Perch Pilot (bottom rung). If you know a "Fact" as interesting as these and send it in ...
And from Hiram Haydn, The Time Is Noon (1948) [combined snippets]:
He could tell himself that she was plenty free and easy with anyone who wore pants, but he knew she liked him particularly, all the same—and he didn't want to get mixed up in that kind of stuff now. Partly because of Ben, but only partly. Maybe he was fancy-schmancy about it, but ever since meeting Sand—
"Meeting" was right. He still couldn't understand it. Chewing the hamburg thoughtfully, he went back over the whole business. ...
I have always thought of fancy-shmancy as being a Yiddish-inflected equivalent of hoity-toity, which Merriam-Webster identifies as "rhyming compound fr. E dial. hoit to play the fool" and dates to 1668.
In The Joys of Yinglish (1989), Leo Rosten argues that the sh- forms are orthographically preferable for Yiddish terms:
IMPORTANT NOTE: I strongly disapprove of using sch or schm instead of sh for English transliterations of Yiddish words. Sch is German; sh is Yiddish—which in fact, uses a single letter (shin) for the sh sound. There is no letter for sch in Yiddish. I use the sh and exile the sch whenever possible.
Nevertheless, in searching for examples of sh- and sch- mock-repetition words in Google Books, I found that the sch- forms tended to be somewhat more common than the sh- forms.