To form a sentence in the past tense, there is no requirement that every verb be in the past tense. You've already demonstrated this. It's "You should have" not "You should had."
In both of these cases, neither the mood of the sentence, nor the tense matter. The issue with given/give/gave is different. Usually, when one verb follows another verb (and the first verb isn't part of a compound tense, e.g. "I have seen"), the second verb will be in the infinitive. For example, "I want [verb 1] to see [verb 2]."
But your examples are somewhat special. The verb let is always followed by a bare infinitive (the infinitive - to give, to drive - minus the to).
You should have let me give
You should let me give
You let me give
If you had let me give
If you will let me give
Here's a BBC article on verbs using the bare infinitive:
let + object + infinitive
Like make, see and hear, let is followed by object + bare infinitive. It cannot be followed by verb-ing:
Let me carry that box of papers for you. It's very heavy.
Why don't you let him walk home by himself from school now? He's eleven years old after all
Let is also frequently used in the expression let's (let us) to introduce a suggestion. Note that negative sentences with let's can be formed in two possible ways:
Let's finish the video tomorrow, shall we? I'm tired and I want to go to bed.
Let's not be late home tonight. It's Monday tomorrow after all.
Don't let's get too stressed about this. I know the car is damaged, but it's only a piece of metal.
We do not normally use let in the passive voice.