Verb constructions or verb patterns as Horny called them are no matter of simple rules. What constructions were adopted in the course of time is to a certain degree arbitrary.
If in doubt you have to look up the common constructions of a verb in the dictionary. See to consider in OALD:

http://www.oxfordlearnersdictionaries.com/definition/english/consider?q=consider

There is no rule that tells you whether the construction is to avoid doing or to avoid to do. Both construction were theoretically possible, but only to avoid doing is used. We can't give a reason why the gerund is preferred. If you don't know the construction you have to look it up.

Of course, you can read a grammar where you will find lists

1 verbs + to-infinitive

2 verbs + gerund

3 verbs + to-infinitive or gerund without difference of meaning

4 verbs + to-infinitive or gerund with difference of meaning.

But such lists give only a selection of the most frequent verbs.