Originally, the words four and forty obviously have the same root, but they were actually pronounced with distinct vowel sounds in many past dialects of English, and still are in some present ones.
These two vowel sounds are those of FORCE and NORTH (to use John Wells's lexical sets). If you pronounce these words with the same vowel sound, it means you have the horse-hoarse merger. This merger is part of the "standard" dialects in England and North America, so most general-purpose dictionaries transcribe these words with the same vowel.
In varieties of English without the merger, the exact way these vowel sounds are distinguished varies among different accents. In old-fashioned British "received pronunciation," words like FORCE had the diphthong /ɔə/ while words like NORTH had the monophthong /ɔː/ (identical to the vowel found in words like THOUGHT). Rhotic (r-pronouncing) varieties of English that distinguish these two vowels usually have a phonetically higher vowel in FORCE (something like /or/ or /oʊr/, which could be characterized as the GOAT vowel followed by the consonant /r/) and a phonetically lower vowel for NORTH (/ɔr/ or /ɒɹ/, which could be characterized as the THOUGHT vowel followed by /r/). Here are some audio samples I found on Youtube of a western Scottish, young female speaker's NORTH and FORCE vowels.
The word four is pronounced with the FORCE vowel, and forty is pronounced with the NORTH vowel.* Apparently, the vowel in forty was shortened at some point in history (the spelling variations may give some clue; another way to find out would be to see what pronunciations were recommended by orthoëpists).
How does this relate to the spelling? Well, in general, the spelling "our" corresponds to the FORCE vowel (in words like source and course, setting aside words like flour or tour), and the spelling "or" corresponds to the NORTH vowel (in words like for and north, setting aside words like word). As the spelling of FORCE itself suggests, words with the FORCE vowel can sometimes be written with "or," but I don't know of any words with the NORTH vowel that are written with "our." So for speakers who distinguish NORTH and FORCE, the spelling "fourty" would not correspond as well as "forty" to the pronunciation of the word. I would guess this contributed towards the eventual standardized spelling without "u".
*Unfortunately, I don't have access to a historical/dialectical dictionary of English pronunciation that I can cite for this, but here are some
relevant links that mention this fact: