I can't find any rule that relates the spelling of words containing "ost" to the pronunciation of the vowel. I don't think it would be of much practical use anyway: since there are so few words with this spelling pattern, it seems easier to just memorize that cost, frost, lost are pronounced with /ɒst/ and ghost, host, most, post are pronounced with /əʊst/. (In fact, because there are so few monosyllabic words that end in -ost, I'm also going to discuss the pronunciations of some polysyllabic words ending in -ost.)
For the most part, the modern differences in pronunciation correspond to historical differences in pronunciation, but a few words seem to have undergone irregular sound changes.
- Old English āst /ɑːst/ seems to regularly correspond to Modern English /oʊst/, as in most < Old English māst, ghost < Old English gāst, oast < Old English āst. There doesn't seem to be any reason why some of these words are spelled with ost and others with oast.
- Old English ost /ost/, or os /os~oz/ contracted with a following dental plosive, seems to regularly correspond to Modern English /ɒst/ (or in different accents, /ɔst/ or /ɑst/), as in frost < Old English frost. (There are historical spellings of frost that show that variant developments also once existed, such as forms with a long vowel or a metathesized form forst.) The final consonant cluster in lost seems to have developed in Middle English; the Old English verb losian "lose" had the past participle (ge)losod.
a unique (as far as I can tell) word from Old English is dost, which has the same /ʌ/ vowel as does, doth, done. This seems to be the result of vowel reduction in a very commonly used verb. (Compare the change in many forms of North American English, but generally not in British English, of the vowel in the stressed forms of common words like was, what, of, because to /ʌ/.)
French ost was generally taken into English as long /oʊst/ (post < French post; host < French (h)o(o)st; toast < French toster; roast < French roster; coast < French coste). The development seems similar to that of words like beast < French beste, feast < French feste, taste < French tast. There doesn't seem to be any reason why some of these words are spelled with ost and others with oast.
- One word with a pronunciation that's hard to explain is cost /kɒst/ (or in different accents, /kɔst/ or /kɑst/), from Old French cost, coust. Two words with a similar etymology, oust (from Old French ouster, oster) and joust (from Old French juster, joster, jouster) are not pronounced the same way as cost. The Oxford English dictionary says that the modern pronunciation of joust, /jaʊst/, is based on the modern spelling, and that the regular development would be /dʒʌst/, as in adjust. Some of the Middle English spellings of cost listed in the OED, such as "coost," "coust" and "coast," seem to suggest that variant pronunciations with different vowels once existed.
- The word accost (from Middle French acoster, accoster) is also pronounced with /ɒst/ (or in different accents, /ɔst/ or /ɑst/), possibly due to association with cost. The Oxford English Dictionary says that the word was formerly associated with coast, and sometimes spelled with oa, so it's possible the pronunciation has changed over time.
- The word provost is sometimes pronounced with /əʊst/ in American English, but the Oxford English dictionary gives /ˈprɒvəst/ as its main British pronunciation. There is also variation in the pronunciation of the first vowel (maybe related to the similar variation in the pronunciation of words like produce and progress). This word has a very confused history, being borrowed into Old English and then "re-borrowed" to some extent later on. The ambiguous spelling probably contributed to the various pronunciations used in Modern English.
Something that I think is interesting is that there is similar ambiguity for words ending in oth: moth, broth and cloth are pronounced with /ɒ/ (or in different accents, /ɔ/ or /ɑ/), , both is pronounced with a long dipthong /oʊ/, and sloth, wroth and troth can each be pronounced either way.