'Which' can certainly be used in indirect speech, as in
- She asked him, "which book do you want to read?"
She asked him which book he wanted to read.
- "It is the new cult of selfishness which has devalued altruism," they complained.
They complained that it was the new cult of selfishness which had devalued altruism.
(Note: in reported speech, 'which' can be replaced with 'that' in certain situations, as in "they complained that it was the new cult of selfishness that had devalued altruism.")
Regarding your specific sentences,
- "Which one is your book?" (she asked him)
She asked him which one his book was (reporting her speech literally) but a better construction would be,
she asked him which book was his.
Remember that you don't need to use the exact same words when putting what somebody said into reported speech.
- "What's the best," (she asked)
She asked what the best was.
(This is, however, an awkward construction because we are not told who she is asking, or what 'best' is referring to. In fact 'which' goes with 'best' more often than 'what' does. When referring to specific things, 'which' is preferable to 'what', as in "which is the easiest language to learn," "which is the best story written by Poe?" and "which is the friendliest canine?" But 'what' is also used with 'best' in other contexts, as in "what's the best explanation for his behavior?" "What would be the best reply to this impertinence?" and "what is the best solution to this problem?")