I understand the phrase's meaning and its usage, but I don't know how it could have come to mean what it does.
Similar phrases like "in a manner of speaking" and "in a sense" make more sense. A sentence like "he's a parrot, in a sense" would mean "there exists an interpretation of the statement that he's a parrot such that the statement holds true". "He's a parrot, in a manner of speaking," would mean that one might speak in a particular colloquial in which "he's a parrot" has an appropriate meaning.
On the other hand, I don't even know how to parse the phrase "so to speak": "so, [if I were] to speak, [I would say ...]"? Or is "so-to speak" a kind of speak? Or does the "so" mean "true", as in "make it so"?