To clarify syntax, I will mark subjects with s in this answer. I will follow the usual convention of marking unacceptable sentences with a preceding *.
The word order in "He knows whats is what" is correct and does not have any subject-auxiliary inversion: the subject (the first what) comes before the auxiliary verb is, which comes before the predicative complement (the second what).
Syntax of interrogative clauses with multiple wh-words
"He knows whats is what" contains two wh-words in the same clause: one as the subject, and one as the predicative complement (as MarcInManhattan rightly notes).
The use of more than one wh-word in the same clause like this is not common in formal language, but when it occurs, more than one wh-word can't be put at the start of the clause, so any additional wh-word(s) beyond the first stay in place rather than undergoing wh-fronting. In this case, the predicative complement what stays in place (or "in situ"). This is normal English grammar for this context; it is not exceptional.
Per Glottopedia's article on wh-in-situ:
in languages with overt movement of one wh-element (like English), the other wh-elements stay in situ.
In the case of whats is what, since the clause has a wh-word as its subject, no subject-auxiliary inversion is possible, whether the clause is embedded or not (compare Whos did it? I don't know whos did it, not *Did whos it? or *I don't know did whos it?).
We do see subject-auxiliary inversion (with do-support if needed) in a non-embedded interogative with multiple wh-words where the first wh-word is not the subject: "What did hes put where?" And when embedded, we would have wh-fronting without subject-auxiliary inversion as "I don't remember what hes put where."
A priori, we might suppose that it should be syntactically possible to front the predicative complement, and leave the subject what in situ, which would result in it inverting with the verb. That would create "What is what?s" and when embedded, "What whats is". But I don't think this actually sounds natural to any speakers, so I don't think that is the structure that is actually used in the expression "know what's what".
In practice, there seem to be restrictions in interrogative clauses with multiple wh-words on which ones can be fronted.
These restrictions seem to be called "superiority effects"; here is a quote from literature studying that topic in a different context:
for speakers that do allow wh-extraction of indirect objects,
the predicted contrast does appear to hold, as shown in (85a-b).
(85) a. ?? Whomi did John give ti what?
b.* Whati did John give whom ti?
(Symmetry in Syntax: Merge, Move, and Labels, by Barbara Citko)
Other analogous examples
Other examples of the same type: "I don't know whos did what" and "I don't know whats goes where" (because the subjects are who and what respectively, the additional wh-words what and where remain unfronted).
Note that this word order is much more acceptable than "*"I don't know what whos did" or *"I don't know where whats goes."