The -er/-re, -our/-or spelling differences (and many more) between English-English and US-English are a symptom of something quite different:
- the most common phoneme in spoken English (South English, North English, Welsh, Scots, N American, Antipodean) is the "lazy vowel", written in IPA as a mirror-imaged lower-case "e", and named as "schwa": but few languages written in Roman-based scripts have a letter for it.
(Side-note: the lazy vowel is present in the Turkish alphabet as a dotless i, with comic side-effect that the upper-case of (dotted) i is a dotted I...)
So if we use ' as a syllable marker, "spectre/specter" is said as SPEC'Tschwa or (depending on the speaker, their mood, whether they are in a hurry, etc.) as SPECT'schwa, irrespective of location.
Likewise honour/honor said as HO'Nschwa or HON'schwa, irrespective of location.
The -er, -re, -or, -our endings are merely desperate attempts (at different places and times) to patch over that missing letter which should be in our alphabet, and which (since Roman times and probably before) should have been.
-er/re and -or/our are merely the two most common cases of this gap being painfully noticeable.
So the answer definitely is: spell the way your audience will expect. Unless you're sure they won't worry about this should-be-trivial matter: but even then, be consistent.
(-: As a Brit who lived in France writing technical documents to be used in the US, I have a greater problem - even now, decades later, each time I want the more common word for "hue", I have to ask "is it -or, -our, or -eur" today? :-)