Good question. I tried thinking up some sort of guideline and found that I didn't really know myself, so I found this article which does a fairly decent job at explaining it and some general rules on how to apply them. Though, like all things in English, if you want to know with any precision how to call people from certain countries, you must memorize them.
With some exceptions, place names ending with:
- ea or silent e get -ean, e.g. Chilean
- a get -n e.g. American
- a vowel get -an e.g. Malian
- otherwise use -ian e.g. Iranian
The use of -ese, according to the article, comes from Italian words borrowed into English from the first traders in the far east and South America, who were Italian.
The article goes onto to say -ic and -er come from Latin via Germanic languages, and are appended to the end of countries ending in -land or island. -ic is usually used to denote "having some characteristic of", e.g. Icelandic whereas -er is usually used to denote a person from that place, thus Icelander.
The other Germanic suffix -ish means "belonging to", and is only used for countries in Europe. The article points out that French and Dutch uses this suffix, but it has been "fused with the base to create a new irregular adjective".
Finally -i comes from Arabic (e.g. Pakistani), and also means belonging to. Almost all countries that get this suffix are in the Middle East, Central Asia (to the north) and North Africa and are Islamic. Countries that don't have had a long history of contact with The West before conversion to Islam.