You only have to use a little bit when it is followed by an adjective. That's a rule. For example, "he is a little bit fool", not "he is a little fool". Another example would be, "speak a little bit louder", not "speak a little louder". It is only grammatical to use it just before an adjective.
