There is no rule. Words with these prefixes have come about through accidents of history. The most usual is "un-", but always consult a dictionary. The following does not really answer your question, but you may find it somewhat useful anyway, especially if you're willing to work to dig some of this out of a dictionary.
Dissertation on "Negative Prefixes" in English.
"a-" is a Greek prefix meaning "not" or "without". It is found almost exclusively with words formed from Greek roots. You can usually spot these by the spellings: "ph", "th", "y", "rh", "chr", "pn", "mn", final "sis" or "ic".
theist / atheist
chromatic / achomatic
rhythmic / arhythmic
symmetry / asymmetry
This prefix is found mostly in scientific terminology, especially in the medical sciences. "agranulocytosis", "apnea", "amenorrhea", "anemia", "apraxia", "amitosis".
However, these are not cases where the prefix was applied to an already existing word. Most people know these words as a single unit. They are unaware that the initial "a" has a separate meaning of its own. These should be learned separately, as there are very few pairs like those cited above.
This prefix is also confusable with the native English prefix "a-", as in "ago", "asleep", "aside", which does not have anything to do with negation.
non- has almost exactly the same meaning as "un-", but is less frequent, and here again the best approach is to learn these separately. It occurs more freely with nouns than many of the other prefixes do. Here are a few common ones:
partisan / nonpartisan
sectarian / nonsectarian
violence / nonviolence
standard / nonstandard
compliance / noncompliance
proliferation / nonproliferation
sense / nonsense
The last listed is the only one where the stress shifts to the prefix.
"in-" is a prefix from Latin, so it is usually seen when the root is from Latin. While native English roots tend to be monosyllabic, Latin roots tend to be polysyllabic. "in-"changes to "im-" before "m", "p", and "b". It changes to "il-" before "l" and to "ir-" before "r". This pattern is quite common with adjectives (derived from Latin).
articulate / inarticulate
polite / impolite
possible / impossible
modest / immodest
legal / illegal
reverent / irreverent
regular / irregular
sanity / insanity
Latin also uses the prefix "in-" in other ways, not necessarily for negation, so caution is advised! For example, "improve" is not the negation of "prove"! Probably the most maddening of these is the word "inflammable", which means the same thing as "flammable", not the opposite! You will sometimes see the word "nonflammable", which is more clearly the opposite of "flammable".
"dis-" is also a Latinate prefix, but it often means more than the simple negation of "un-". With verbs it may imply some action (often of removal) employed to create a negative state or the absence of something. The difference is usually more obvious in the past participle. Usually the form with "un-" cannot even be used as a verb.
arm / disarm (remove weapons from)
unarmed - not carrying a weapon
disarmed - having had one's weapon( s) taken away
infect / disinfect (remove possible sources of infection)
uninfected - not having an infection
disinfected - having had possible sources of infection removed
qualify / disqualify (remove from competition or consideration)
unqualified - not having the proper qualities or qualifications
disqualified - judged to be unqualified; having been removed from consideration
The prefix "de-" is also sometimes used in the sense of removal, forming verbs from nouns: "defrost", "delouse", "dethrone", "devein", "defrock", "declaw", "deice".
Sometimes the positive form has a prefix which is removed before the negative prefix is added. "encourage / discourage" "consonant / dissonant"
With some dictionary work you should be able to discover the difference between the words in these groups as well. They are rather curious, not to say pathological, examples.
interested / disinterested / uninterested
prove / disprove / improve
integrate / disintegrate / segregate / aggregate
assemble / disassemble / dissemble
distinguish / distinguishable / indistinguishable
distinguished / undistinguished
claim / disclaim / unclaimed
able / unable / disabled
trust / distrust / trustworthy / untrustworthy
cover / uncover / discover
"un-" is the native English prefix for negation, but it combines freely with nonnative roots as well. It is the most used prefix of its kind. It is used with verbs, adjectives, and adverbs, but also sometimes with abstract nouns -- not with concrete nouns ("*an unchair", "*an unbowl"). It can indicate simple negation (adjective) ("happy / unhappy") or it can indicate reversal of a process (verb) ("lock / unlock"). In this latter role, the prefix "de-" is sometimes used instead.
able / unable || tidy / untidy || cooperative / uncooperative || safe / unsafe ||
helpful / unhelpful || grateful / ungrateful || likeable / unlikeable || suitable / unsuitable || kind, unkind
Whenever there is a common word which is the opposite, the "un-" form does not exist: high / low (*unhigh, *unlow) fast / slow (*unfast, *unslow). But speakers sometimes mistakenly use such forms as "unthaw" for "thaw" (freeze / thaw, *unfreeze / *unthaw) or "unloosen" for "loosen" (tighten / loosen, *untighten, *unloosen).
pack / unpack || dress / undress || screw / unscrew || wind / unwind || tie / untie || roll / unroll || veil / unveil || cover / uncover
(Note how many of these form phrasal verbs with "up", e.g., dress up, wind up, tie up, roll up, cover up.)
code (encode) / decode || activate / deactivate || hydrate / dehydrate || humanize / dehumanize || escalate / de-escalate || brief / debrief
Here are a few curious examples. Get out your dictionaries! These could be challenging!
rail, derail; plane, deplane; attach, detach, unattached, detached, undetached; compose, decompose.
Does "derail" mean "remove the rails from"? If you have all planes removed from the runways, do you deplane the runways? Can you "rail" something? Can you "plane" something? What are the different implications of "attached" and "undetached"? Don't they mean the same thing (because two negatives (un, de) make a positive)? Is decomposing really the reversal of composing?