Before 1600, the OED gives citations where forty is spelled in various ways, but never with just an "o" vowel:
- feuortig, feortiȝ, fuwerti, uourty (the "u" is really a "v"), fourty, fourthi, fourtie
This might possibly mean that there was some actual diphthong leading to these spellings; since most of these spellings occurred before there was any standardization, it is hard to tell either way.
In 1600, there are three citations, and, interestingly, all of them have just "o":
- 1602 Contention Liberalitie & Prodigalitie i. iv. sig. B2, "Cham sure chaue come, vorty miles and twenty."
- a1642 J. Suckling Poems (1646) 37, "And there did I see comming down Such folks as are not in our Town Vorty at least, in Pairs."
- 1698 J. Fryer New Acct. E.-India & Persia 94, At the end of their Quarentine, which is Forty days.
(We can ignore the "v/f" alternation — something was apparently going with the voicing at the beginning of this word, but it probably has no bearing on the vowel following it.)
Aside from one citation in the 1700s that uses fourty, everything else from then on is written as forty.
One can only guess the reason for this change (at least with the information that I have) — whether it was pronunciation shifting or just orthographic simplification. But I might have an explanation for why this spelling took hold so swiftly in the 1600s: the Bible. The King James edition of the Bible was a major influence on the standards in English spelling. The KJV Bible was published in 1611 (begun in 1604), and (since I happen to have a KJV corpus handy) I see that there are 158 tokens with the spelling forty in KJV and 0 tokens for fourty.
So, even if the spelling of forty was following the whim of a handful of publishers, it got into the King James Bible, and that was that.