I think this question has many easy answers in informal contexts.
By way of example,
He doesn't read Tom Clancy: he only reads high-brow books.
I disagreed with her about whether or not that novella qualified as serious literature.
We love Game of Thrones but don't really think of the books as really deep literature.
All of these sentences are various levels of informal, and the last one is something which someone might say but would be unlikely to write.
They all also carry the desired meaning.
However, if you are writing in a formal context, I don't think there is a simple word or phrase that encapsulates this idea.
Depending on your needs, merely using the word "literature" may carry all of the connotations you desire.
It lends a bit more gravitas than "book", which is a bit pedestrian by comparison.
If you feel that the distinction is not sufficient, you might write "high-caliber literature".