FumbleFingers explains the meaning well, here's some extra notes. I also noticed a fair number of citations from the navy.
The original phrase was "no more chance than a cat in hell without claws" (with variation) and (taking a different tack and searching for "hell without claws" rather than "cat in hell" turns up some with a few mis-scanned "cat") I can find no earlier occurrence than the 1759 The life and real adventures of Hamilton Murray, written by himself linked in FumbleFingers' answer:
From the same search, we can find this phrase in Francis Grose's 1788 A Classical Dictionary of the Vulgar Tongue:
A humorous poem in The Gentleman's Magazine, for April, 1792, written from the point of view of a cat in hell, sent to his master on earth, has a footnote that claims a (tongue-in-cheek?) origin:
The first that doesn't mention claws is 1783's A Letter to Sir Phil. Jen. Clerke
In the 19th century, John Bellenden Ker "endeavoured to retrace the original form of the words which I believe to have then duly conveyed the sense of the phrases of the above category. By applying the sound of the words which constitute the modern phrase to others which it fitted in the Low Saxon stage of our language, I have always found a sense, corresponding with that conveyed by the form under which they are now disguised, to be the result of the experiment. The following pages contain the proofs of this test."
Well, I'm not entirely convinced by the "proof", but here it is from 1834's An essay on the archaeology of our popular phrases, and nursery rhymes:
And by 1837 Bellenden Ker produced another edition with a whole page of explanation, which I won't reproduce here, but his digging resulted in this "origin": so that the existance of villainy contains within itself a clear proof there must be a hell.
The 1892 Journal of American Folklore reports the phrase used to mean lack of peace rather than lack of chance:
There 's no more peace here than for a cat in hell without claws. (Ohio.)
However, still the most common meaning was to have no chance, to be helpless or defenceless.