I understand the phrase when it is used literally in this example.
He's ill and doesn't know what he's saying.
I perceive it to mean that a person is struggling with an illness that prevents him from thinking clearly and understanding his words. However, there have been times where the usage seems more vague to me, particularly when it is directed at an able-minded or sound-minded person.
He doesn't know what he's saying.
I don't think you know what you're saying.
Okay. I think there are some possibilities.
A person uses the phrase as a personal attack on the other person's intellect.
A person uses the phrase to discredit the other person.
A person is implying that the target person is oblivious to a connotation or perception of the target person has said.
A person uses the phrase to expose evidence that the target person is a dilettante.
Perhaps, the phrase means each one in different contexts, or perhaps, it means a combination of them?