Per my first comment to the question, "guesstimates" from Google Books: one person's word against another's -4990, one person's word against another - 3120. So you could say the "full" version "wins". But they're only estimates which are often wildly inaccurate - on a more specific search where I can actually check them all, it's another - 27, another's - 21.
The logical position - since it's always "your word against his" (not "your word against him"), obviously OP's version should also use the possessive.
But in practice many (perhaps even most) writers apparently choose to ignore that. And it's a pretty safe bet they're even more likely to do this in speech (people tend to write more "correctly" than they actually speak). And I rate the "grammar" of speech higher than that of logic and textbooks.
I personally find the "logically/grammatically" correct version unnecessarily cumbersome. There's no case for claiming the meaning could possibly be affected - we all know what it means, and it would be perverse to suppose "you against him" means anything different in such contexts than "your word against his word".
TL;DR: Pedantically speaking, you "should" use the possessive apostrophe. But I wouldn't bother.
As it happens (and as I just commented to @David's answer) I noticed "I asked her for a list of Charles' friends" when watching Missing, 1982 last night. Only one "s" was enunciated, which seems perfectly normal to me. If the context already strongly implies the possessive, why throw in another awkward consonant? So many people don't, it just seems pointless to claim they should.