I agree with elias that there is no simple rule for this. Something that supports this viewpoint is that a number of words have shown variation over time, or still show variation: a famous example is "inalienable" vs. "unalienable".
But it may be possible to give fairly accurate rules for certain subcategories of -able adjectives.
If the -able word is based on a monosyllabic verb, use un-.
If you can remove the suffix -able from the adjective and get a monosyllabic English verb, the adjective almost certainly is negated with un- and not with in-.
I would guess that there are more than a hundred examples of words that follow this rule: to start with, you can consider unthinkable, unspeakable, unbearable, unstoppable, unflappable, unshakable, unforeseeable, unsalable, unquenchable, unbridgeable, unworkable, unlovable, unlikable, unwearable, unbreakable, unchainable, unplayable, unwinnable.
Un- is possible even when the verb is of French or Latin origin, as in untouchable, untreatable, unusable, unnotable.
I only know of four common exceptions to the monosyllable rule: incurable and impassable (for which the un- alternatives have negligible usage), immovable (which is much more common than unmov(e)able), and insolvable (which is currently less frequent than unsolvable). H. W. Fowler's Modern English Usage (first published 1926; republished 2009) mentions these four in a longer list of in-_-able words (-able, p. 5; for more information about Fowler's list, go to the next section).
Insuitable in place of unsuitable seems to be obsolete, although it can be found in dictionaries and some old documents.
If the -able word is not based on a monosyllabic verb...
An -able adjective that is related to a verb of more than one syllable may take either un- or in- as the negative prefix: un- is generally more common and productive, but I don't know of any particularly simple rule that tells you which prefix to use for all words in this category. You can try to reason from the etymology, as described in tchrist's answer: an -able word built on a verb with Germanic etymology will take un- as a rule. The etymological criterion can be used to rule out in- for unanswerable, unutterable, unforgettable.
However, etymology usually can't be used to rule out un-, because un- is used with many -able words derived from Latin or French.
Fowler (1926, cited in the previous section) gives a list of around 100 -able words where he recommends forming the negative with in-, and says to use un- for any word not on his list.
Minor rules based on the spelling of the end of the word
If the word ends in -kable, use un-.
I don't know of any exceptions to this in modern usage, but the words covered by it are mainly a subset of the words covered by the previous rule. We do also get unremarkable, unmistakable, unrebukable, unattackable. Inattackable seems to have once existed, but I think it's pretty much never used anymore.
If the word ends in -cable, you can use in-.
There are words ending in -cable that can be negated by un- (e.g. uneducable, although apparently some people prefer the sound of ineducable). But I haven't found any -cable adjectives that cannot be negated with in-.
If the word ends in -yable, use un-.
As with -kable, most examples are monosyllables, but we also have undestroyable, unemployable and unenjoyable. In-/im- may be found in unassimilated French words that are occasionally used in English such as impayable, incroyable.
If the word ends in -onable, use un-.
There aren't so many words that end like this, but I think there are enough to identify this as a pattern. Many of these words end more specifically in -ionable or -tionable. Examples: unquestionable, unexceptionable, unmentionable, unobjectionable; unconscionable, unfashionable; unseasonable, unreasonable,
unpardonable. "In-" is not always completely impossible in this context; "inconscionable" exists, but is much less common. "Infashionable" has been used occasionally in the past (it's in the OED), but is now obsolete.