I get the sense that the interjection nay is seen as outdated and used only for humorous effect. Is this assumption true, or is it still acceptable in serious writing?
As to the meaning of 'no'
- (archaic or dialect) no: nay, I must not think thus.
But it still has another meaning:
- or rather (used to emphasize a more appropriate word than one just used): permission to build the superstore will take months, nay years.
I would look at the context of the writing to determine whether or not it seems 'jocular', as you suggested, but I wouldn't be surprised to find this word used in so-called "serious writing."