Louise Pound wrote two brief articles (1930, 1931) that, together, elucidate the uncertain etymology of 'stir' in the sense of "a prison or jail".
In the 1930 article, titled "The Etymology of STIR 'Prison'", Pound states
…the history of the term has not been carefully traced. Perhaps, in the absence of links pointing toward the older forms, its ancestry cannot be certainly supplied. It seems to me far likelier, however, that the word, if coming from the Old English, is to be associated, not with the verb styrian, agitate, but with the noun stēor (stīer, stȳr) defined in Old English dictionaries as meaning (1) steering, guidance (2) rule, regulation (3) restraint, discipline, check, correction. The phonetic development from either stēor or stȳr is normal, if we keep in mind the sixteenth-century coalescence of -er, -ir, -ur, -or. The shortened vowel of a Middle English stēr or stīr might have arisen out of compounds like, stēorness, stȳrness, correction, discipline, or stēorlēas, without restraint, stēorwirþe, deserving reprobation, etc. Or compare sir, shortened from sire. On the semantic side the development of stir from Old English stēor seems even more probable than on the phonetic side.
I was well-satisfied with this explanation, particularly considering a previous finding in Hotten's 1860 A dictionary of modern slang, cant, and vulgar words:
STIR, a prison, a lock-up; "IN STIR," in jail. Anglo Saxon, STYR, correction, punishment.
And again in the much-maligned Charles MacKay's 1877 The Gaelic etymology of the languages of western Europe:
STIR (Slang).—A prison.
In stir, in jail; Anglo-Saxon, styr, correction, punishment. [Hotten's 1860] Slang Dictionary.
Gaelic.— Stuir, management, direction, control; to direct, to steer; whence "in stir," under control (of the prison authorities).
There remained to discover why (or how) OED disregarded the rather obvious derivation from stȳr. However, I had observed in passing that Pound followed the 1930 article with another article, in 1931, titled "The Etymology of STIR 'Prison' Again", so I sought that out with the hope of resolving the question. In that second article, Pound writes
[My previous] suggested etymology seems likely enough, and I am unable to find that it has been brought forward hitherto. [We now know Pound's suggested etymology was brought forward by Hotten in 1860, and quoted by MacKay in 1877.] An objection to it, however, may be found in the gulf of time elapsing between the disappearance of the Old English word, in printed monuments, and the emergence of the underworld word in late modern times.
There is another possibility that should be taken into account, that suggested by the familiar tradition of the relation between nineteenth-century slang expressions and the language of the Gypsies. Among the words used by the English Gypsies … are stardo (past participle) 'imprisoned,' staripen 'prison,' staromengro 'prisoner.' … A derivation of the monosyllabic noun stir from the initial syllable of these formations is perhaps less acceptable phonetically than that from the Old English, but it is easily possible and it seems more acceptable chronologically. The connection between slang and the speech of the Gypsies is not so close, it may be, nor the debt so great, as was believed in the nineteenth century. Nevertheless, some words in widespread usage — e. g., the familiar pal — are credited to the English Gypsies, and stir may belong in the same category. Certainly, whether the noun be of native or gypsy origin, a better spelling would be ster, dissociating it to the eye from the verb stir 'agitate.'
The Gaelic stuir (as suggested in MacKay, 1877) might provide the chronological link Pound sought between Old English and the "emergence of the underworld word in late modern times". That connection seems especially likely considering that (a) Romani derives in part from Gaelic, and (b) the parallel development of 'sturabin' (a form of the Romani staripen) is also offered in MacKay 1877:
STURABIN (Slang).—A prison.
Gaelic.— Sturr, a rock; sturrag, a pinnacle, a high tower; sturragach, pinnacled, having towers like French prison; abhuin, water; whence sturrabin [MacKay seems to be proposing a Gaelic compound of sturragach and abhuin], a rock in the water, inaccessible.
I note for the sake of completeness that derivation of 'stir', "a prison", from 'Start', a nickname for Newgate prison, seems to be founded on a slender phonological resemblance and the otherwise fanciful association of 'stir' with a transparently jocular entry in the 1823 Slang, by John Badcock (pseudonym Jon Bee):
Start (the) — Newgate is thus termed, par excellence. But every felon-prison would be equally a start….
The 1860 Hotten also contains an entry for 'the start'. No mention is made of 'stir' in that entry, as is also the case with the entry in the Badcock's 1823 Slang.
START, "THE START," London,— the great starting point for beggars and tramps.