"Worst nightmare" is the right one. "Worst", as well as milder adjectives of the same type, can be thought of as subtracting "goodness points" or "adds badness points" to something, not "negation" as in a sign flip.
If you want to think of it like math, think of the noun as having a value number, say, e.g. a "field trip" has a "value number" of +5. (Arbitrary -- this is for illustration.) Positive numbers are good, negative numbers are bad. Now suppose the adjective "worst" is applied, to make it "worst field trip". This subtracts (or "adds negative/'bad' points"), say, 10 points (also arbitrary), turning it to -5, now something bad.
Now consider the word "nightmare". It is already worth "negative points", say -5 points. Add the adjective "worst" on the front, to make "worst nightmare". It does the same thing as in the previous example. It subtracts 10 points, making it now -15 points, thus taking something already bad and making it even more bad.
Of course, English is not a precise count of numerical points, but the gist is that "worst" applied to "nightmare" is actually not really doing anything different from what it is doing when applied to "student". I do not think there is any adjective in English that actually acts like a mathematical negation in this analogy, i.e. which when applied to something "good" flips the "sign" to make it now "bad" and when applied to something "bad" flips the "sign" to make it now "good" with no change in the form of the adjective at all.