Here I don't understand the grammar of this sentence--
I never heard you say that.
Why not 'I never heard you saying that' or 'I never heard of you saying that'?
I'm not sure why you don't understand the grammar, since it's precisely the same as the other two. They just have slightly different meanings.
This means that you never heard the entire event of the person saying the thing. You still could have heard the beginning or end of them saying that or something that might have been part of that.
This means that there was never a time when the speaker heard the person in the process of saying the thing (whether or not they completed the act, whether or not the beginning was witnessed).
If you heard the person saying something that might have been part of that, you could truthfully make the first claim, but you could not be sure the second claim was true.
This means the speaker never heard any reports of the person saying the thing. The first two could be true even if you heard someone else claim they said that, but this one would be false.