You got it right in your title. A word that is not spelled phonetically (that is, a word that's pronounced differently than it's spelled) is commonly called a non-phonetic word.
I couldn't find any authorities which use this term, but it's found all over the web. For example, this site defines non-phonetic words as "words that aren't pronounced like we'd expect."
This term is natural given certain senses of phonetic:
agreeing with pronunciation
phonetic (Oxford Dictionary)
(Of a system of writing) having a direct correspondence between symbols and sounds.
We can easily imagine this latter definition being broadened to include in its domain words.
You can also find many discussions of phonetic languages versus non-phonetic languages.
A phonetic language is defined here as a language in which
The letters in its alphabet consistently correspond to the same sounds and form reliable patterns of pronunciation. If you know the rules, you can spell any word you hear. The relationship between spelling and pronunciation is strong.
The mapping between spoken sounds and written letters is much looser and inconsistent in English than in other languages like Spanish and Russian. Thus, English is regularly brandished as an example of a non-phonetic language. For example, here, here and here.
Given the above, it seems like non-phonetic word is exactly what you're looking for.
Unfortunately, this term would apply not only to extreme cases like victuals and colonel, but probably also to words like island and weigh and many others (especially loan words). If you only want to include extreme cases where the pronunciation isn't even mildly predictable from the spelling, you might use the term highly non-phonetic word.