Saying "Whom did you give the book?" is rare seems to me an understatement; I haven't found any actual examples of sentences like this. (As far as I know, who can always be used instead of whom, so I'm treating "Who did you give the book" the same way in this answer.) Despite the rarity, I have found conflicting statements from grammar experts about whether it is definitely ungrammatical (which may reflect a similar variance in native speakers' intuitions.) All of them agree that is is at the very least infelicitous for many speakers.
As ChongDogMillionare said, the fact that we can say
You gave him the book.
(using a noun phrase "him") as well as
You gave the book to him
(using a prepositional phrase "to him") is called dative alternation. We can use the noun phrase form in some complex sentences, such as passives (He was given the book) and interrogative sentences where the direct object is fronted (What did you give him?).
So, your question is basically: can we use the noun phrase variant in an interrogative sentence with a fronted indirect object? (Whom did you give the book?)
My intuition is that you can't. But I don't know why not! In the comments, Ben Voight gave an example sentence You gave whom the book? and suggested it was grammatical. I don't know if I agree; it sounds awkward to me, but I don't know if that means it's ungrammatical. If it is grammatical, and Whom did you give the book? isn't, then the issue must be with the fronting specifically. There are other syntactic differences between questions with fronting/wh-movement and those with wh-in-situ, like the use of an auxiliary (did in this case).
Here is a relevant quotation that says the sentence is ungrammatical for this reason:
Wh-movement distinguishes between direct and indirect object: *Who did
you give a book?; What did you give John?
(Essays on Anaphora, by H. Lasnik)
It's a convention in linguistics to use a preceding asterisk to mark a sentence as ungrammatical. But I also found this:
Wh-movement of the indirect object from the V NP NP frame is generally
regarded as infelicitous. Fodor (1978) explains this with reference to
a processing restriction she terms the XX Extraction Constraint. In
parsing a string in which the indirect object has Wh-moved, the parser
would have difficulty detecting the gap since its expectation that an
NP will follow the verb is in fact met by the direct object NP. [...]
?Who will the children show the new toy?
("Prior Linguistic Knowledge and the Conservatism of the Learning Procedure: Grammaticality Judgements of Unilingual and Multilingual Speakers", by Helmut Zobl)
The wording here doesn't seem quite as strong ("infelicitous") and a question mark is used, which indicates less certainty that it is ungrammatical than an asterisk would.
And here is another quotation that implies that some speakers find it grammatical, even though many others don't:
Testing superiority effects in English double object constructions is
also tricky, since for many speakers wh-movement of indirect objects
is impossible even if the indirect object is the only wh-element.
However, for speakers that do allow wh-extraction of indirect objects,
the predicted contrast does appear to hold, as shown in (85a-b).
(85) a. ?? Whomi did John give ti what?
b.* Whati did John give whom ti?
(Symmetry in Syntax: Merge, Move, and Labels, by Barbara Citko)
Here, ti represents an imperceptible trace left behind before the indirect or direct object is moved.
I found a source that points to some further discussion:
Another asymmetry[...] has often been noticed in the literature on
double object constructions (cf. Hudson, 1992; Larson, 1988; Oehrle,
(11) a. What did she send her sister t?
b. ?*Who did she send t a book?
c. Her sister was sent t a book.
d. ?*A book was sent her sister t.
[...]Wh-Movement can extract the direct object in a double object construction, whereas NP movement can extract the indirect object. Neither of these processes is problematic. However, the reverse pattern of extractions is problematic for many speakers (cf. Hudson, 1992, who discusses variability in judgements here.)
("Constraints on Argument Structure," by Kenneth Hale, Samuel J. Keyser, in Syntactic Theory and First Language Acquisition: Cross-linguistic perspectives Vol. 1 Heads, Projections, and Learnability by Barbara Lust, Margarita Suñer, John Whitman)
I found a PDF of the relevant paper by Hudson, "So-called “double objects” and grammatical relations."
I also found an interesting example sentence that seems similar but is treated as grammatical, although it has an additional interrogative word:
Who did you give what at Christmas?
(Pragmatic Syntax, by Jieun Kiaer)