The quotation actually occurs as a part of this exchange between Miss Innes (the narrator of the story) and Mr. Jamieson (a detective), following a remark by the detective that might be construed as humorous but that has agitated Miss Innes:
"I have always heard," I said drily, "that undertakers' assistants are jovial young men. A man's sense of humor seems to be in inverse proportion to the gravity of his profession."
"A man's sense of humor is a barbarous and a cruel thing, Miss Innes,"
he admitted. "It is to the feminine as the hug of a bear is to the scratch of—well;—anything with claws. Is that you, Thomas? Come in."
Mr. Jamieson's comparison of "a man's sense of humor" to "the feminine [sense of humor]" has a couple of complications. First, a person may suppose that being hugged by a bear is less unpleasant than being scratched by, say, a puma. And second, one "thing with claws" is ... a bear; so one possible comparison that Mr. Jamieson's comments haven't excluded is "the hug of a bear" versus being clawed by a bear.
To make sense of the dialogue, I think we first have to understand that the hug of a bear is not at all a pleasant thing. It is in fact one stage of being mauled by a bear. So if we compare the bear hug to, say, getting scratched by a house cat (as Hot Licks's comment above suggests), we would have to conclude that the scratch is much less painful and much easier to tolerate.
On the other hand, the detective does not characterize the feminine sense of humor as benign. According to him, it expresses itself through scratches, just as the masculine sense of humor expresses itself through the crushing weight of an aggressive embrace. In some sense, Mr. Jamieson's broken-off comparison of men's and women's senses of humor is itself a kind of failed joke, at the end of which he turns his attention to Thomas with a kind of welcoming relief.
In the end neither sense of humor seems especially innocent or sincere. Both function as weapons appropriate to the creature that wields it, and both call for wariness on the part of anyone in the vicinity who may be exposed to them.