As RegDwigнt's answer mentions, verbs that form agent nouns in -or usually have the form of Latin perfect passive participle stems.
Latin perfect passive participle stems characteristically end in t or s. So in practice, verbs that form agent nouns in -or almost always end in -t(e) or -s(e), and form agent nouns in -tor or -sor specifically.
There are exceptions, but not very many.
A large proportion of the exceptions only occur in legal English, where -or is used more than in regular English (often as a counterpart to -ee). For example, "deliveror" and "settlor" are basically only found in legal contexts; in other contexts, the spellings "deliverer" and "settler" are used.
Exceptions that are not restricted to legal English include conquer-or, sail-or, purvey-or, survey-or, counsel-(l)-or, and vend-or (the last has a less common but accepted variant vend-er).
There are other -or words that don't end in -tor or -sor, but most of the rest aren't built directly on an English verb as a base. For example, donor, emperor, tailor, juror don't refer to people who "done", "emper", "tail" or "jure".
Ending in -t(e) or -s(e) is (mostly) necessary for a verb to form an agent noun in -or, but it certainly isn't sufficient. Sven Yargs left a comment mentioning "disrupter" as an example of an agent noun that is commonly spelled with -ter.