- What courses is everyone taking?
This is technically grammatical, even though it may sound odd to a non-negligible amount of people. "What courses" is not a grammatical subject in this sentence: this is made clear by comparing it to the sentences "What courses is he taking?", "What courses am I taking?", and "Everyone is taking those courses." The pronoun "everyone" is always singular as a subject, so the grammatically correct verb form is the third-person singular "is" (agreeing with the subject "everyone").
- What courses are everyone taking?
This is correct only if you would accept "are" in a sentence like "Everyone are taking those courses".
Sentences with inversion can be tricky to parse, which means the grammatically correct option may not be immediately apparent and might not sound right. Feel free to restructure these kinds of sentences if the correct version sounds wrong to you (a comment by Edwin Ashworth suggests the alternative "What are the courses that everyone is taking?").
The use of "what" or "which" doesn't make a difference to the verb agreement. In this context, either interrogative adjective is possible, but which is more formal: the Oxford English Dictionary says what as an interrogative adjective is
Used in asking the identity of a choice made from an indefinite set of alternatives. Also (now chiefly colloquial): used in asking the identity of a choice from a definite set of alternatives (= which adj. A.I.3).
There is a separate question about what vs. which.
Other similar questions: