Just recently, I heard someone on TV use the phrase "I kid of course". (I think it was The Big Bang Theory but I'm not very sure). Any way, is this phrase grammatically correct?
Yes. One might quibble about not having a comma between "kid" and "of," but you are transcribing spoken dialogue.
"You are attacked by a tiger, what do you do?"
"I run, of course"
In case this is the cause of your confusion, the work kid here is the verb meaning to talk (especially tell a lie) jokingly. Nothing to do with children; it does not mean "I am a kid".
So I kid means "I am joking."
I can't really explain why it is in the present tense instead of the present progressive, but this is very often seen specifically with kid (or jest or joke), and hardly ever with other verbs. Perhaps it comes from a historical utterance of the phrase, or such usage in another language. (I somehow associate it with old Jewish comedians.)
Of course the present progressive form "I am kidding, of course." is perfectly acceptable and and common.
It's usually said immediately after an unbelievable-sounding statement, and followed by a clarification (often starting with "But seriously...").