This way of using nor is indeed somewhat archaic.
Normally, nor requires that another negative has been employed somewhere previously. In your Odyssey quote, for example, we might imagine something like this:
Two vultures sat, one on either side, and tore his liver, plunging their beaks into his bowels. He could not rise himself, nor could he beat them off with his hands.
However, in older texts (especially, in my experience, older translations of Greek texts), nor could sometimes ‘piggyback’ not only on an expressed negative, but also on anything that implies a negative somewhere in the situation.
In this case, the despondent impuissance of Tityos creates such a situation. There is no explicit negative, but it is easy to imagine that the situation as a whole functions as a kind of ‘abstract negative’ that nor can be used to emphasise. Simply using not would just make it a plain statement that these birds were tearing out his innards and he couldn’t get rid of them—but using nor emphasises that the birds did not stop, he did not like it, it was not good … the whole thing was, basically, negative.