To your first question, the answer is 4: probably both. You're right, the modifier Chinese is ambiguous, but only in terms of what language the author uses. (Note that ambiguity in headlines is not necessarily something to reject. Eradication of ambiguity often requires the sacrifice of attention, and attention-grabbing is paramount in a headline.) In my opinion, Chinese author is not ambiguous about the nationality of the author. As slight confirmation of this, Mo Yan sounds Chinese, so I'd be comfortable inferring that he is a native. As to the language he writes in, it's very likely to be Chinese as well, but not absolutely necessarily.
To the second question, the obvious way is to say what you mean in as many words: Mo Yan, Chinese native and author of Chinese literature, wins Nobel prize. You can see why that wouldn't "take". You could drop the "native" part, since the name and the fact that he writes in Chinese could be enough to go on: Mo Yan, author of Chinese literature, wins Nobel prize or simply Author of Chinese-language novels wins Nobel Literature Prize, if you're willing to drop the name in favor of conciseness.
But here it becomes obvious that a newspaper headline sometimes needs that trivial ambiguity to stay more interesting. If you need to dispel ambiguity, read the article.