The final consonant of the verb is relevant, indirectly, because of the etymology of the two endings. However, there is no simple rule for the spellings, because there are some complications due to both etymology and analogy.
Latin agent nouns end in -tor or -sor
As RegDwigнt's answer mentions, verbs that form agent nouns in -or usually have the form of Latin perfect passive participle stems.
Latin perfect passive participle stems characteristically end in t or s. So in practice, verbs that form agent nouns in -or almost always end in -t(e) or -s(e), and form agent nouns in -tor or -sor specifically. In the context of Latin, the suffix is often given as -tor rather than as -or.
Examples:
- actor (Latin and English), from the Latin verb ago ("do, act"), perfect passive participle stem act-
- possessor (Latin and English), from the Latin verb possideo ("have, hold, possess"), perfect passive participle stem possess-
You can see that these two words have the same spelling in modern English as in Latin. For complicated reasons, the corresponding English verbs have the form of the Latin perfect passive particle stems: act and possess.
-or preceded by any other consonant comes from French
There are agent nouns in English that end in -or preceded by a letter other than t or s. The main origin of this type of -or agent noun is French. In Old French, the t in the Latin ending -atorem (as well as some other similar endings) was lost by regular sound change, and the syllables merged, causing it to become -or/-our/-ur (Modern French -eur). This was taken into Middle English as an agent-noun suffix -our.
In Modern English, this ending came to be unstressed and its vowel became reduced, making it sound identical in English pronunciation to the -or ending found in Latin words, and its spelling has therefore been simplified in most cases to -or.
A large proportion of -or agent nouns only occur in legal English, where -or is used more than in regular English (often as a counterpart to -ee). For example, "deliveror" and "settlor" are basically only found in legal contexts; in other contexts, the spellings "deliverer" and "settler" are used.
Some -or agent nouns that are not restricted to legal English are conquer-or, purvey-or, survey-or, counsel-(l)-or, and vend-or (the last has a less common but accepted variant vend-er).
sail-or is an example of the French suffix being used on an English base, the verb sail.
There are other -or words that don't end in -tor or -sor, but most of the rest aren't built directly on an English verb as a base. For example, donor, emperor, tailor, juror don't refer to people who "done", "emper", "tail" or "jure".
There seems to be sporadic use of the form -or instead of -er simply as a means of forming words with a specialized meaning, sometimes as part of technical terminology or jargon.
The Oxford English Dictionary entry for the word "sailor" describes it as "An altered spelling of sailer n., probably assimilated to tailor, in order to distinguish the designation of a regular calling from the unspecialized agent-noun. The differentiation, however, does not appear in our early examples, and was not fully established before the 19th cent."
In electronics, the form "expandor", as well as a derived term "compandor" (from compressor and expandor), has apparently been used.
It's hard to rule out -er as a potential spelling for an agent noun
Ending in -t(e) or -s(e) is (mostly) necessary for a verb to form an agent noun in -or, but it certainly isn't sufficient. Sven Yargs left a comment mentioning "disrupter" as an example of an agent noun that is commonly spelled with -ter.
The suffix -er is highly productive and is used to form nouns with a range of meanings. I don't think any rule based on meaning will be reliable as a guide to using -er vs. -or. To give counterexamples to some alleged rules that I've sometimes seen:
Words ending in -er can be animate (runner, worker, speaker, reader) or inanimate (washer, dryer, circuit breaker, holder). Words ending in -or also can be animate (administrator, director) or inanimate (elevator, insulator, ventilator).
Words ending in -er can be names of professions (teacher, writer, baker, publisher, typographer, undertaker), as can words ending in -or (sculptor, legislator, conductor).