As @John notes above, any way of expressing the past allows reported speech.
we always use this word when talking about the past.
But for more conceptual clarity, can't we use the following hypothesis:
He said that I was a very stupid man-- This sentence does not reflect that he repeatedly told you that you were a stupid man.
He used to say that I was a very stupid man-- (Past) More than one instance of him telling you that you were a stupid man but he no longer says that you're stupid.
E.g. When Joshua was a child, he used to climb trees. (Now he doesn't climb trees.)
When the base form of the verb is used.
- She didn't use to swim before noon. (Now she does swim before noon.)