The most important thing to remember is to always use /ʃən/ after a vowel: for example, in words like condition or preparation – using /tʃ/ here would sound very unnatural. One word that is a weird exception is equation, where most speakers have a voiced consonant /ʒən/ for some unclear reason.
The pronunciation /tʃən/ is only required after an /s/ (in words spelled with -stion); /ʃən/ is generally used after any other consonant.
The only complication I can think of that applies to more than one word is the pronunciation after /n/, in the sequence -ntion. For many speakers, the contrast between /ntʃ/ and /nʃ/ is neutralized in some contexts, in particular after a stressed syllable (this is analogous to the "prince-prints merger"). So some speakers might pronounce a word like attention with [tʃən]; but these speakers might also have [tʃən] as a possibility in the word tension.
The difference between the pronunciation of the suffix in words like relation (or abstraction) on the one hand, and words like congestion on the other, is actually rather old (it predates the introduction of these words to English) and while it isn't reflected in the French and English spellings (which are based on the Latin etymology), it is reflected in the spelling of equivalent words in languages with more phonemically transparent writing systems:
Spanish: relación, abstracción, congestión
Italian: relazione, astrazione, congestione
Of course, the "rule" I outlined does not apply to all words that just end in the letters "tion" in English: the most notable counterexample is the word cation. However, I can't think of any counterexamples to my rule when dealing with the actual suffix -tion.
Note: I used the example words relation and abstraction rather than reflection because the latter word has alternated with reflexion in the past, giving it a more complicated history.