This is actually much simpler than one might expect, if one only paid attention to spelling.
The name of the phenomenon is Assimilation,
and it's a very natural process that occurs in every human language.
If a prefix ends in a consonant, and then is attached before some other consonant,
then the consonant at the end of the prefix is going to change in contact.
That's called assimilation; and in fact assimilation is an example of assimilation,
because assimilation comes from Latin ad 'toward' + simil 'same'; i.e, 'become similar'.
But the D at the end of ad changed to an S, making a double SS
(which was probly pronounced as a double /s/ in Latin; this happened in Latin).
In the examples above, you'll find that an underlying final consonant (say B in sub-)
will appear before vowels, but before /k/ (C in Latin) it becomes /k/ (suc-), before /p/ or /t/ it becomes /p/, before /r/ it becomes /r/, etc. There are individual rules for lots of prepositions and prefixes (which constitute the same class of morphemes, but define different uses for them).
As for a general rule, it's generally the following consonant that determines the fate of the consonant at the end of the prefix. That's because by the time you've gotten to the end of the prefix/position you know what it is and you don't need to hear the end, so you slur over it to get to the beginning of the next morpheme.
This happens all the time, I repeat. But with Latin prefixes, the assimilations were already done and fixed in place before they came to English, so they're not productive anymore and have to be learned by rote (or by learning Latin, as it used to be done).