This is an example of a past subjunctive being used to express unreality. At first look it doesn't seem like there's anything "unreal" in the sentence, but consider the difference between the following (from here):
It's time to go to bed
It's time you went to bed
The first expresses that bed time is right now, while the second implies that bedtime should already have happened. Because the sentence is not a simple statement of fact ("this is the time for us to talk") but a statement of obligation ("we should talk/should have talked by now"), the subjunctive is used. It's not necessary to phrase it this way - "It's time to have a talk" is equally grammatical - but the subjunctive construction conveys that extra implication.