All that he did was at thy instigation, thou viper; for I thou thee, thou traitor.
In "I thou thee", "thou" is a verb. The relevant definition in the OED is:
trans. To address (a person) with the pronoun thou (or its equivalent in another language).
(The quote in your question is one of the examples listed for this sense, in fact.)
It's really the same pattern as "Don't 'honey' me!" which you may have actually heard in real life.
To understand the reason this was an insult, one must be aware that "thou" at this time was a familiar pronoun of the second person singular while "you", the plural, had come to be used in the singular for people owed a certain measure of respect. (Compare French "tu" and "vous", if you're familiar with French.) Therefore, to "thou" someone whose relationship to the speaker would ordinarily call for the use of "you" was a show of grave disrespect.
The comparison with French has been made:
(thou) Tu es une vipère. En effet je te tutoie, tu es un traître!
(you) Vous êtes une vipère. En effet je vous vouvoie, vous êtes un traître!
But you may also compare with German:
(thou) Du Schlange! Ich duze dich, du Verräter!
(you) Sie Schlange! Ich sieze Sie, Sie Verräter!
The cool thing with French and German is that you can explicitly see that they actually have a verb for saying "thou" as opposed to "you".
The verb for "thou" ("tu" in French, "du" in German): "tutoyer" in French and "duzen" in German.
The verb for "you" ("vous" in French, "Sie" in German): "vouvoyer" in French and "siezen" in German.