We have a proverb in Bengali, if I translate it directly into English, it emerges as:
Who is in there in the temple? I did not eat the banana!
Meaning in the temple banana is used for prayer to the God. So stealing banana from temple is very bad. But the person who steals always wants to pretend that he is not responsible for that. But he has the fear in his heart. So if anybody enquires anybody, other than him, as to who has stolen the banana, the thief speaks the first(albeit he is not asked!) that he did not do anything indicating he is the thief! The whole proverb, in a short, signifies that who has done the wrong, speaks the first when asked to others, revealing he is the doer. So what might be the proverb in English or expression to express the same meaning?