It should be use to.
Why? Because you're using a did(not) in your sentence. The general rule is that when you use 'did' in a sentence, the tense used is present.
For example (though in both cases, the intention becomes slightly different due to emphasis)
I did go to the party. vs I went to the party.
You worked with Clara vs You did work with Clara
Similarly with your example:
You used to work with Annie, right? vs Didn't you use to work with Annie?
