The final consonant of the verb is relevant, indirectly: because of the etymology of the ending *-or*, it is usually found after ***t*** or ***s***. However, there is no simple rule for *-or* vs. *-er* that always gives the right ending, because there are some complications due to both etymology and analogy.

## Latin agent nouns end in *-tor* or *-sor*

As RegDwigнt's answer mentions, verbs that form agent nouns in *-or* usually have the form of Latin perfect passive participle stems. 

Latin perfect passive participle stems characteristically end in **t** or **s**. So in practice, verbs that form agent nouns in *-or* almost always  end in *-t(e)* or *-s(e),* and form agent nouns in *-tor* or *-sor* specifically. In the context of Latin, the suffix is often given as *-tor* rather than as *-or.* (There is an unrelated Latin suffix *-or* that forms abstract nouns from verbs, such as *error, rigor, terror*).

Examples:

- *actor* (Latin and English), from the Latin verb *ago* ("do, act"), perfect passive participle stem *act-*
* *possessor* (Latin and English), from the Latin verb *possideo* ("have, hold, possess"), perfect passive participle stem *possess-*

You can see that these two words have the same spelling in modern English as in Latin. For complicated reasons, the corresponding English verbs have the form of the Latin perfect passive particle stems: *act* and *possess.*

## *-or* can follow other consonants in words from French

There are agent nouns in English that end in *-or* preceded by a letter other than *t* or *s*. The main origin of this type of *-or* agent noun is French (specifically, the Anglo-Norman variety that was spoken in England after the Norman conquest). In Old French, the *t* in the Latin ending *-atorem* (as well as some other similar endings) was lost by regular sound change, the *a* turned into a schwa (first spelled *e* and then eventually lost), and as a result the whole thing ended up as *-or/-our/-ur* (Modern French *-eur*). This was taken into Middle English as an agent-noun suffix *-our*. A simplified example: Latin *gubernator* (accusative *gubernatorem*) became Anglo-Norman French *gouvernour*, Middle English *governour,* Modern English *governor.*

In fact, there was extensive spelling variation and some confusion with other suffixes of similar meaning already in French, so the Oxford English Dictionary lists not only *gouvernour* but also *governor, governour, governur, guvernur, governer, guverner* as possible spellings of the Anglo-Norman word, and *couernour, couernoure, gouernowre, gouernur, gounour, gouvernour, guuernere, guuernour, governore, gouernere, governowre, gouerner, gouernor, gouernour, gouernoure, governoure, governour, governer* as potential spellings in Middle English.

The ending *-our* was pronounced /uːr/ in Middle English and was at least sometimes stressed (Chaucer rhymes *governour* with *flour* 'flower'¹) but in Modern English, this ending came to be unstressed and its vowel became reduced, making it sound identical in English pronunciation to the *-or* ending found in Latin words. Along with the preexisting variation and the analogical influence of Latin *-or*, this made it easy for the spelling *-our* to be simplified in most cases to *-or.*

- A large proportion of *-or* agent nouns only occur in legal English, where *-or* is used more than in regular English (often as a counterpart to *-ee*). For example, "deliveror" and "settlor" are basically only found in legal contexts; in other contexts, the spellings "deliverer" and "settler" are used.

- Some *-or* agent nouns that are not restricted to legal English are *conquer-or, purvey-or, survey-or, counsel-(l)-or,* and *vend-or* (the last has a less common but accepted variant *vend-er*). These all originate from (Anglo-Norman) French nouns, but can be reinterpreted in modern English as suffixed derivatives of the English verbs *conquer, purvey, survey, counsel, vend.*

- *sail-or* is an example of the French suffix being used on an English base, the verb *sail.*

There are other *-or* words that don't end in *-tor* or *-sor,* but most of the rest cannot be interpreted as being built directly on an English verb as a base. For example, *donor, emperor, tailor, juror* don't refer to people who "done", "emper", "tail" or "jure".

There seems to be sporadic use in English of the spelling *-or* instead of *-er* simply as a way of forming words with a specialized meaning, sometimes as part of technical terminology or jargon. 

- The Oxford English Dictionary entry for the word "sailor" describes it as "An altered spelling of sailer n., probably assimilated to tailor, in order to distinguish the designation of a regular calling from the unspecialized agent-noun. The differentiation, however, does not appear in our early examples, and was not fully established before the 19th cent."

- In electronics, the form "[expandor](https://www.thefreedictionary.com/expandor)", as well as a derived term "compandor" (from *compressor* and *expandor*), has apparently been used.


## It's hard to rule out the spelling *-er*

Ending in *-t*, *-te,* *-s* or *-se* is mostly necessary for a verb to form an agent noun in *-or* (with exceptions, as discussed above), but it certainly isn't sufficient. 

- [Sven Yargs left a comment mentioning "disrupter"](https://english.stackexchange.com/questions/4733/what-s-the-rule-for-adding-er-vs-or-when-forming-an-agent-noun-from-a-ver/486896#comment803421_137523) as an example of an agent noun that is commonly spelled with *-ter* (although the spelling *disruptor* also exists). *Adopter, corrupter, tempter, respecter* are other difficult-to-predict common spellings in *-ter.*
 
- *User* and *passer* are always spelled with *-ser.*

The suffix *-er* is highly productive and is used to form nouns with a range of meanings. I don't think any rule based on meaning will be reliable as a guide to using *-er* vs. *-or.* To give counterexamples to some alleged rules that I've sometimes seen:

- Words ending in *-er* can be animate (*runner, worker, speaker, reader*) or inanimate (*washer, dryer, circuit breaker, holder*). Words ending in *-or* also can be animate (*administrator, director*) or inanimate (*elevator, insulator, ventilator*).

- Words ending in *-er* can be names of professions (*teacher, writer, baker, publisher, typographer, undertaker*), as can words ending in *-or* (*sculptor, legislator, conductor*).

----
¹"Appollo, god and governour / Of every plaunte, herbe, tree and flour"—"The Franklin's Tale"