I would say that "rejectance" is not a widely recognized word. As you mentioned, it doesn't seem to be in any of the commonly available dictionaries; I also wasn't able to find it in the Oxford English Dictionary, which sometimes contains obscure or obsolete words that don't have entries in other dictionaries, or in the Corpus of Contemporary American English (COCA). It is fairly easy to understand approximately what "rejectance" means/would mean (some abstract noun related to the verb reject), but it sounds odd to me. I would not recommend using it as an interchangeable synonym of "rejection"; if you don't have a particular reason for using "rejectance", I would recommend just using "rejection".
There is no fixed list of "legitimate" or "proper" English words, so it's not clear what it means to describe a word using one of these adjectives. All the major modern dictionaries take a "descriptive", not a "prescriptive" approach to the language, and there are certain characteristics of English usage that make compiling an accurate list of "legitimate" English words (as opposed to merely dictating an artificially restricted list) a formidably difficult, if not impossible, task. (For more detailed explanations of this issue, see What does the phrase “a real word” mean?)
Some English suffixes are highly productive. For example, -er to create agent nouns from verbs and -ness to create abstract nouns from adjectives.
Although these suffixes cannot be attached to absolutely any word of the appropriate part of speech, they can be attached freely enough that you wouldn't necessarily expect a dictionary to include all of the words that can be formed with these suffixes and sound natural to a native English speaker (even if that speaker has never heard that particular word before).
However, the suffix -ance, used to form a nominalization from a verb or from an adjective in -ant, is not so productive. If a word with this suffix can't be found in the dictionary, it is probable that it will sound at least a bit odd to a native English speaker due to its rarity. But it is not completely unproductive either.
Etymologically, the suffix -ance originally comes from Latin words (transmitted partly through French), but it does appear in some words that were formed in English and not Latin. The Oxford English Dictionary (OED) says
As, in many cases, the Old French vbs. themselves, as well as their derivatives in -ance, were adopted in English (e.g. appear -ance, assist -ance, purvey -ance, suffer -ance), the suffix became to a certain extent a living formative, and was occas. used to form similar nouns of action on native vbs., as abid-ance, abear-ance, forbear-ance, further-ance, hinder-ance, ridd-ance, etc.
Now that we've gone over some background info, let's look at "rejectance" specifically. As far as I have been able to ascertain, there is no French word "rejectance" or "rejetance".
In Latin, "reiectantia" (or "rejectantia") would I believe be theoretically possible as a derivative of reiectare, the frequentative of the verb reicere, but I haven't found any evidence of this actually being attested as a word used in Latin.
So it seems the word would have to have been formed in English.
One thing that makes it more difficult for a (relatively) newly formed word like "rejectance" to attain common usage is the phenomenon of "blocking". The main idea here is that, if a word already exists with a certain meaning, people are less likely to use another affix to derive a synonym for the preexisting word.
The theory of "blocking" would predict that "rejectance" would be more likely to spread if it had some kind of specialized meaning that distinguished it from the preexisting noun "rejection". It also might be more likely to gain ground with non-native speakers who have not yet internalized the word rejection as the noun corresponding to the verb reject.
As you point out, anology is another thing that can affect the formation of words. It's true that reject and accept are in many contexts antonyms, and therefore have a semantic connection that might influence someone to form their nouns in similar ways.
A look through the Google Books results for "rejectance" suggests that both of these factors are relevant to the use of this uncommon formation. It seems to turn up particularly often in proximity to the word "acceptance", and in jargon, which often has a need for coining words with specialized, somewhat unintuitive meanings. As I am not familiar with any of the relevant fields of study, I can't really say what it means in particular disciplines, but Phil Sweet's answer seems to explain one of the jargon uses of "rejectance".