I think commas are typically placed after closed parentheses and
within quotation marks. This creates a dilemma when all three are used
The idea behind parentheses is they are not part of the sentence, and this is the first thing to bear in mind. If you don't, you are surprised that the verb does not agree with the sense of the whole sentence. You find an example here
Rule 3. Parentheses, despite appearances, are not part of the subject.
Example: Joe (and his trusty mutt) was always welcome.
If this seems awkward, try rewriting the sentence:
Example: Joe (accompanied by his trusty mutt) was always welcome.
but if you want to be sure, just rewrite it like this :
Joe () was always welcome.
The same article says that
Rule 4. Commas are more likely to follow parentheses than precede
Incorrect: When he got home, (it was already dark outside) he fixed
Correct: When he got home (it was already dark outside), he fixed
But there is also anoter reason: your example:
You may like snelms (or "snail helms,") which are etc.
Would become: You may like snelms which are .... and it would be incorrect, since commas are necessary here:
... there certainly are times that a comma must be used before the
word. When which is used at the beginning of a non-restrictive
clause, it must have a comma. Another comma is placed at the end of
that clause. A non-restrictive clause gives added information that is
not necessary. The meaning of the sentence can be understood without
it. The example given in Grammarly's Handbook
is: That box of apples, which I picked this morning, can be used to
make the pie. We don't need to know that the apples were picked this
morning to understand that we can use the apples to make the pie.
As to the irrational rules in American style guide, they have a trivial technical origin:
Compositors―people who layout printed material with type―made the original rule that placed periods and commas inside quotation marks to protect the small metal pieces of type from breaking off the end of the sentence. The quotation marks protected the commas and periods. In the early 1900s, it appears that the Fowler brothers (who wrote a famous British style guide called The King’s English) began lobbying to make the rules more about logic and less about the mechanics of typesetting. They won the British battle, but Americans didn’t adopt the change. That’s why we have different styles.
and, if you are not "...writing for publication at any U.S. publishing house that follows either Chicago or AP style,...", you'd better follow logic, anyway.