"It had been time for bed" needs some help from adverbials:
(S1) "By the time he finished dinner, it had already been time for bed for quite a while."
This sentence (S1) means that bed time definitely started before he finished dinner.
But the original sentence uses "was" to allow for the possibility that the finishing of dinner was simultaneous with bedtime:
(S2) "By the time he finished dinner, it was (already) time for bed."
There are two possibilities inherent in the word "by": (a) before, and (b) at the same time. For example, "by 3 o'clock" means "either before 3 o'clock or at 3 o'clock", or in other words "no later than 3 o'clock".
"By" has the same meaning in sentence S2, which says that it became time for bed either before he finished dinner or at the very moment he finished dinner. S2 clearly means that he did not finish dinner before it became time for bed.
Adding "had" gives us the original sentence:
(S3) "By the time he had finished dinner, it was time for bed."
This does not change the meaning of "by" and does not change the two inherent possibilities. Sentence S3 still means that he had not finished dinner before it became time for bed.
Changing "finished" to "had finished" in a "by the time" clause does not invert the meaning of "by" or make the sentence mean that finishing dinner came before time for bed.
Notice that "he did not finish dinner before bedtime" and "he had not finished dinner before bedtime" also exclude the same possibility.
Conclusion: Past perfect does not automatically change the order of events. We use past perfect to express either that one event happened before another or that one event did not happen before another.
One more example:
(S4) "He was caught before he had run half a mile."
This means that he did not run half a mile before he was caught. In fact, the use of "had caught" in S4 suggests that, because he was caught, he never completed that half-mile run.