Let us go
is a correct construction in the English language and definitely not:
Let we go
However, the question is: since us is an object-case pronoun, what is the subject of this sentence?
It's an imperative, and in English the imperative tense only is possible in the second person.
So grammatically, the subject has to be "you".
It can't be interpreted literally; it's an idiom whose meaning isn't really an imperative that you do something, but a suggestion that we do something.