Is there a specific term which can be used for books glorifying evil or the villain?
The answer depends largely on your intent in referring to the book, or on the writer's intent.
If you want to emphasize the way a book disregards or exists in conflict with common morality, then a convenient and effective phrase description is "transgressive" or "an example of Transgressive Literature".
If you want to emphasize the role of the main character as an Anti-hero, then the terms "anti-heroic" or "morally ambiguous" are serviceable (but not elegant).
If you want to emphasize that a book as an attempt to change our perception of a character or action as villainous then you might say "subversive" or "an Apologia". Usually these would be applied in non-fiction contexts.
Without further understanding the intent of usage, it's impossible to narrow it down further, so here are a few other candidates: Malevolent, Malicious, Misanthropic, Amoral, Immoral, etc.
I think you mean antihero. Not really a villain but a character who is unconventional and veers away from the traditional blueprint of a hero
The antihero or antiheroine is a leading character in a film, book or play who lacks some or all of the traditional heroic qualities, such as altruism, idealism, courage, nobility, fortitude, and moral goodness. ...
Otherwise the Villian Protagonist
The difference between an anti-hero and a villain protagonist is that the anti-hero often has the role of a hero but with darker sides.
This list is meant to help list books with characters that have the role of what would normally be considered an antagonist, someone you normally wouldn't cheer for. Maybe they have sympathetic sides, maybe they don't but the important thing is that it's the main character of the story.
Examples of Villain Protagonists are (including video game villains): Beetlejuice; Darth Vader; Slender Man; Sasuke Uchiha; Hannibal Lecter; Carrie White; Godzilla; Chucky; Venom (Marvel); Bowser...
If you don't need to go too dark, you may be thinking of a picaresque novel:
picaresque of or relating to an episodic style of fiction dealing with the adventures of a rough and dishonest but appealing hero
The picaresque hero is dishonest and may be somewhat villainous, but has interesting adventures and enough redeeming qualities that readers can enjoy the tale without being too appalled.