a- and an- are used in words with Greek origin, (meaning no, absence of, without, lack of, not in said language). Other Greek prefixes are ana-, (meaning again, up, on, above, c.f. anabaptist literally meaning "re-baptist") and anti- (meaning against, c.f. antidemocrat).
Anyway, the thing is that as language is a living thing with lots and lots of users (especially in the case of English, where there also are a lot of different cultures, further contributing to the confusion), and therefore, rules like this tend to be mixed up (especially when creating new words or combining already existing ones). The rule of thumb is that, as previously stated:
un- is used in words of Germanic origin,
in- in words of Latin origin (and generally in words from other Latin-derived languages, like Spanish or French),
- a- and an- is used in words of Greek origin.
Prefixes like de-, dis- and dys- come from Latin or Greek and have a slightly different meaning.
Note that un-, in- and a-/an- have the same origin, but they have taken very different ways into our language of today.