The OP is right in assuming that, if the comma were to be inserted, then the sentence would be roughly equivalent to 'Thanks, but no, I refuse the offer; thank you once again'. That is, however, not quite what was intended by the original sentence.
The original sentence has the tone of a relatively casual dismissal of whatever was offered. Such a casual dismissal is incompatible with thanking the offeror very profusely, which would be conveyed by 'Thanks . . . thank you once again'.
The key to understanding the original sentence is to note that thanks by itself can sometimes be used to mean 'no, thanks', if it is accompanied by a suitable gesture, or the context otherwise makes it clear that the offer is being rejected. In 'Thanks, but no thanks', the first thanks is such a thanks of rejection. The speaker then, to remove any ambiguity about the rejection, and perhaps to emphasise it, proceeds to say 'but no thanks'. The way to understand the original sentence is thus as something like 'Thanks, and by that I mean "no, thanks"'.
Now, in 'by that I mean "no, thanks"' we still have the comma. In that rephrasing of the original sentence, the quotation marks surrounding the 'no, thanks' make it easy to see that these two words are to be read together, notwithstanding the comma. However, if we were to remove the quotation marks, as was done in the original sentence, and keep the comma, it would appear that no and thanks perform some separate, independent roles in the sentence; it would be difficult to read them together as one building block of that sentence. It is to avoid that difficulty that the comma is omitted in the original sentence, even though, comma would otherwise be included in 'no, thanks'.