If you're talking about a bookshelf, you might use both. For general talk about any sort of shelf, I'd tend to go with on, because gravity makes it hard to sit things beside or under a shelf (without the help of additional shelves), and physics makes it hard to place a book inside (in) the actual shelf.
It's possible to say that the book is in the bookshelf (that is, within the confines of the unit as a whole), meaning that it is part of the overall collection of other objects, books, and so on which are enclosed by the bookshelf. This might particularly apply to an enclosed bookshelf, such as one with hinged glass doors covering the front (arguably a cabinet).
The book is in the bookshelf somewhere.
However, it is more common to say that the book is on the bookshelf when you are referring the specific location of the cook on the shelf, where it is currently physically resting, regardless of what is elsewhere in the shelving unit.
The book is on the shelf over there.