Monosyllabic words are not a problem: you know when to double a final consonant before adding a suffix, since the vowel in the word is always stressed:
rob+ed/ing/er = one final consonant preceded by one stressed vowel > double consonant: robbed, robbing, robbery, a robber
Words made of more than one syllable can be a bit of a problem:
The stressed syllable in the verb "to infer" is the second one.
So the rule which goes "before adding a suffix to a word, double its last consonant if the word ends in ONE CONSONANT only, preceded by ONE STRESSED VOWEL only" applies to the verb forms (past simple, past participle, -ing form).
infer+ed/ing = one final consonant preceded by one stressed vowel > double consonant: he inferred that… / he has inferred that… / inferring that…
But in the noun "an inference", the stress is moved to the first syllable. A case a bit similar to the noun "an increase" opposed to the verb "to increase". So, before you add the noun suffix –ence, you have a word "infer" ending in one consonant all right, but preceded by one UNSTRESSED vowel this time.
infer+ence = one final consonant preceded by one unstressed vowel > single consonant: an inference
Whereas in words like "to differ" and "a difference", the stressed syllable remains the same, the first one, and, as a result, the last consonant is doubled neither in the verb forms, nor in the noun:
differ+ed/ing/ence = one final consonant preceded by one unstressed vowel > single consonant: one thing differed from another / one thing has differed from another / differing from something / a difference
However, there are exceptions to this rule!
Michael Swan, in Practical English Usage, 2nd edition, 1995, Oxford University Press, wrote, on page 558:
"In British English, we double -l at the end of a word after one vowel letter in most cases, even in unstressed syllables.
travel > travelling; equal > equalled
In American English, words like this are normally spelt with one l: traveling.
other exceptions:
Consonants are sometimes doubled at the end of final syllables that are pronounced with full vowels (e.g. /æ/), even when these do not carry the main stress.
kidnap > kidnapped; handicap > handicapped; worship > worshippers (US also worshipers); combat > combatting or combating
Final -s is sometimes doubled in focus(s)ing, focus(s)ed, bias(s)ed, and similar words.