But since "use to" can't be used here,
Who has told you that?
We didn't use to go swimming a lot, but now we love it.
Is just fine. After a supporting do we use an infinitive, not a conjugated verb.
Any google search for didn't we use to will find you articles about this, like for example this one from LearnEnglish:
Did you use to go out with my sister?
Did they use to own the company?
Didn't we use to go to the same school?
Basically use behaves the same way that any verb does when we switch to do-support: (for completeness, we only look at the simple past here because of the idiomatic usage of use).
He walked to school. -->walked in simple past
He didn't walk to school. -->supporting do in simple past + infinitive
He used to walk to school. -->used in simple past
He didn't use to walk to school. -->supporting do in simple past + infinitive
All that said, in the linked topic it is shown that I didn't used to, although grammatically baffling, has been gaining popularity, probably because of the similar pronunciation. So people decided to always write used without having to think about it.
That we end up with a sentence with to conjugated verbs for one subject in one clause, soit. We need exceptions, right? :)