The words four and forty obviously have the same root if you go back far enough, but they were actually pronounced with distinct vowel sounds in many past dialects of English (and still are in some present ones). Four was pronounced with a "long o" sound, while forty was pronounced with a "short o" sound.* Apparently, the vowel in forty was shortened at some point in history (the spelling variations may give some clue; another way to find out would be to see what pronunciations were recommended by orthoëpists at different times). This is analogous to the difference in the pronunciation of the "ow" in know and knowledge.
How does this relate to the spelling? Well, in general, the digraph "ow/ou" is not used in English to represent a "short o" sound. The word knowledge that I just mentioned is an exception, but there are not many other words like it. So for speakers who pronounce forty with a "short o" sound, the spelling forty reflects the pronunciation better than the spelling fourty. I would guess this contributed towards the eventual standardized spelling without "u".
More details about these two vowel sounds
In John Wells's system of lexical sets, the vowel sound in four (historically a "long o" followed by "r") is called the "FORCE" vowel, while the vowel sound in forty (historically a "short o" followed by "r") is called the NORTH vowel.
If you pronounce these words with the same vowel sound, it means you have the horse-hoarse merger. This merger is part of the "standard" dialects in England and North America, so most general-purpose dictionaries transcribe these words with the same vowel.
In varieties of English without the merger, the exact way these vowel sounds are distinguished varies among different accents. In old-fashioned British "Received Pronunciation," words with the FORCE vowel were pronounced with a centering diphthong transcribed /ɔə/, while words with the NORTH vowel were pronounced with the monophthong /ɔː/ (identical to the vowel found in words like THOUGHT). Wells says /ɔə/ merged into /ɔː/ for most British speakers during the early twentieth century.
Rhotic (r-pronouncing) varieties of English that distinguish these two vowels usually have a phonetically higher vowel in FORCE words (something like /or/ or /oʊr/, which could be characterized as the GOAT vowel followed by the consonant /r/) and a phonetically lower vowel in NORTH words (/ɔr/ or /ɒɹ/, which could be characterized as the THOUGHT or LOT vowel followed by /r/). Here are some audio samples I found on Youtube of a western Scottish, young female speaker's NORTH and FORCE vowels.
*Here are some
relevant links that mention this fact: