As RegDwigнt's answer mentions, verbs that form agent nouns in *-or* usually have the form of Latin perfect passive participle stems. 

Latin perfect passive participle stems characteristically end in **t** or **s**. So in practice, verbs that form agent nouns in *-or* almost always  end in *-t(e)* or *-s(e),* and form agent nouns in *-tor* or *-sor* specifically.

There are exceptions, but not very many. 

- A large proportion of the exceptions only occur in legal English, where *-or* is used more than in regular English (often as a counterpart to *-ee*). For example, "deliveror" and "settlor" are basically only found in legal contexts; in other contexts, the spellings "deliverer" and "settler" are used.

- Exceptions that are not restricted to legal English include *conquer-or, sail-or, purvey-or, survey-or, counsel-(l)-or,* and *vend-or* (the last has a less common but accepted variant *vend-er*).

There are other *-or* words that don't end in *-tor* or *-sor,* but most of the rest aren't built directly on an English verb as a base. For example, *donor, emperor, tailor, juror* don't refer to people who "done", "emper", "tail" or "jure".

Ending in *-t(e)* or *-s(e)* is (mostly) necessary for a verb to form an agent noun in *-or,* but it certainly isn't sufficient. [Sven Yargs left a comment mentioning "disrupter"](https://english.stackexchange.com/questions/4733/what-s-the-rule-for-adding-er-vs-or-when-forming-an-agent-noun-from-a-ver/486896#comment803421_137523) as an example of an agent noun that is commonly spelled with *-ter.*