There was a similar question five years ago: https://ell.stackexchange.com/questions/11611/what-do-you-care
The answer from that question pointed out that "Why Do You Care?" is a genuine question while "What Do You Care?" is more of a dismissive way to talk to someone, saying they don't or shouldn't care about something.
In the case of the Feynman book, I think it is not fair to characterize it as dismissive. I could imagine those eight words being spoken in a positive way.
Here's an example of a friendly usage of that phrase: // "I'm worried that my classmates think I'm a loser," said Jim. Bob gave Jim a friendly look. "Why worry? What do you care what they think?" //
I agree it is a very unusual usage. I don't hear it often.