How I remember being told over and over how to spell tongue! I didn't understand it then; I don't understand it now. What evolution might put a silent "ue" at the end of a word?
In Old English, a language from the West Germanic family, the standard spelling was tunge (wiktionary) and the corresponding pronunciation was /ˈtʊnɡe/ "tun-ghe" (/ʊ/ as in foot). In other words, the final "e" was not mute but clearly pronounced. For instance in present-day German, you would say "Die Zunge" /ˈtsʊŋə/ (tsung-e). In both cases the final unstressed e is actually already a reduction of an older final a still present in many other cognates.
The spelling of Middle English is strongly influenced by that of Old French in which the un sequence is uncommon. It was thenceforth often replaced by on which led to such spellings as tonge or tounge ('ou' being the French spelling of /u/ and /ʊ/) . Which can be compared to contemporary alternative spellings longe and lounge for the word long.
With the evolution of the pronunciation of long vowels in Modern English, these spellings became highly misleading. "Tounge" for instance could be read /taʊndʒ/ (as in lounge). As a result, such spellings as tonghe or tongue were preferred. The fact that the simpler spelling tong was not preferred (in spite of the existence of such words as long) is probably a sign that the final e was not completely muted yet.
Finally, the tongue variant probably overtook the tonghe option as a result of an analogy with such words as L. lingua or languages.
Also note that on the other side of the English Channel, the Dutch word followed a very similar path. From Old Dutch tunga it successively evolved as Middle Dutch tunge, tonghe and present-day Dutch tong.
This is the explanation offered by the OED regarding the spelling and the etymology of the word tongue.
[OE. and ME. tunge wk. f. = OFris. tunge, OS. tunga (MLG., LG. tunge, MDu. tonghe, Du. tong), OHG. zunga, zunka (MHG., Ger. zunge), ON. tunga (Da., Norw. tunge, Sw. tunga), Goth. tuggô:—OTeut. *tungôn-, held to be cogn. with L. lingua tongue, for older *dingua (as lacrima:—dacrima: see tear n.1).
The natural mod.Eng. repr. of OE. tunge would be tung, as in lung, rung, sung (and as the word is actually pronounced); but the ME. device of writing on for un brought in the alternative tonge with variants tounge, townge; app. the effort to show that the pronunciation was not (tundʒ(ə) led to the later tounghe, toungue, tongue, although it is true that these hardly appeared before final e was becoming mute, so that its simple omission would have been equally effective. The spelling tongue is thus neither etymological nor phonetic, and is only in a very small degree historical.]
The spelling comes from the French, and indicates that the g was once pronounced -- in other words, it wasn't always just an engma (ŋ). However, the word tongue doesn't come to us from French; the Old English and some Middle English used tunge. It appears that the French style of spelling won out somewhere in the 14th century, but since that time the pronunciation has changed. For that matter, the sorts of French words, like langue, that influenced the English spelling have also (mostly) lost the g sound over time.
A case of pronouncing the word the way it originally was, even when the spelling changed.
Interesting to note that this word was originally spelt "tounge":
1380, John Wyclif, English works (1880 edition by the Early English Text Society) He schal make his tounge cleue faste to þe roof of his mouþ.
1536, Register of Riches, in Edward Ledwich, Antiquitates Sarisburienses; or, the history and antiquities of old and new Sarum (1771)
Having ... two white Leopards and two dragons facing them as going to engage, their tounges are done in curiousest wyse.
The "e" was always silent, and I suppose they just moved the "u" to the back when they standardized spelling. This doesn't mean that all "-ue" endings are silent. It was spelt "tongue", but it is pronounced "tunge", because it came from "tunge":
O.E. tunge "organ of speech, speech, language,"
This doesn't mean that all "-ue" endings are silent, it's just that in this special case, it is pronounced the way it was originally spelt, even when the spelling changed.
One case of "-ue" being pronounced is "Montague", pronounced: