I don't know of any useful rule for when O is pronounced as /ʌ/.
There aren't that many words where O in a stressed syllable is pronounced as /ʌ/, so I think it's most practical to just memorize the pronunciation of each word with this spelling pattern using some method like flashcards. Masha Bell's "Improving English spelling" blog gives the following list of words:
The main alternative spelling for short u is o
Among, brother, comfort, company, compass, front, Monday, mongrel, monk, month, mother, nothing, other, smother, son, sponge, ton, won, wonder, worry
Sometimes o accompanied by redundant letters
Above, come, does, done, dove, glove, love, monkey, none, shove, some, tongue, blood, flood
In 21 words, the irregular spelling of the short /u/ sound is
not followed by a doubled consonant (as in funny, runny)
Colour, covenant, cover, covet, covey, dozen, govern, honey, money, onion, oven, shovel, slovenly, stomach, thorough.
Couple, courage, cousin, double, nourish, trouble.
Sometimes the letter o is redundant (country, southern, touch, young)
or is accompanied by further irregularities (enough, hiccough, rough, slough, tough, one, once).
There may be some words not on this list, but I would guess that the total number of words where O = /ʌ/, excluding inflected and derived forms like sponging and brotherhood, would be under a hundred.
There are a few sub-patterns—e.g. -other and -ove—but there are words with these spellings that have other pronunciations (like bother and grove) so they can't be used to reliably indicate which words are pronounced with /ʌ/.
The sub-patterns might be used as "warning signs" to indicate which words might plausibly be pronounced with /ʌ/: e.g. if the O doesn't occur in one of the following contexts (oth, on, om, mo, wo, ov, ough, the digraph ou) it's quite unlikely that it is pronounced as /ʌ/, although not completely impossible (as far as I can tell, the words color and dozen don't follow any identifiable sub-patterns for the use of O as a spelling of /ʌ/).
Another piece of evidence that there is no obvious rule: even native speakers show variation or change over time for certain words. For example, conduit and conjure were once commonly pronounced with /ʌ/, but now usually have /ɒ/. The words hover, hovel and grovel currently show variation between /ʌ/ and /ɒ/ (I don't know the relative frequencies, or how they differ between dialects). The uncommon word wont can be pronounced with various vowels, among them /ʌ/. The exact etymology of donkey is unclear, but some scholars think that it comes from the word dun, and that the currently standard pronunciation with /ɒ/ is a spelling pronunciation.