You ignored the usually's etc in the original post.
In terms of "the doctor" and "the speeches", the definite article "the" doesn't indicate that those people/things were talked about before. It just means that people already know that they exist.
After squirrels and balloons, I feel the key word that you missed is the "usually". There other reasons for ...
With this exact wording, no. It would be ungrammatical. "I have spent..." is a relative clause to "Many's the hour," which is interestingly already inverted itself:
Many is the hour is like saying Fair is my beloved!
And the relative clause should follow as:
Many is the hour (that) I've spent listening to his fatuous ideas.
With its adverbial synonym "...