Consider unbelievable versus its synonym incredible, and you will find what there is to be found of an answer here.
The general tendency is to use un‑ on Germanic words or any generic English term, and to use in‑ (possibly mutated; see next paragraph) for words of Latin origin. This is not hard and fast, however; there are of course exceptions.
Also, the Latinate in‑ prefix comes in other forms, like illegible, immutable, irreducible. Those also count as in‑ versions, not un‑ versions.
Similarly, ‑able is the more general ending, but -ible also frequently occurs. In fact, you will find that ‑able and ‑ance tend to go together, just as ‑ible and ‑ence tend to go together.
Here again there is an etymological explanation: whether it derived from either a Germanic word or from a first-conjugation Latin verb (so Germanic words or Latin ‑are verbs yield ‑able type endings) on the one hand, or whether it was instead from another Latin conjugation (so ‑ere verbs and such yield ‑ible type endings) on the other.
But as before, there are notable exceptions.