A lovely and fascinating question!
As you point out, sic erat scriptum, "thus was it written," would literally only apply when quoting from a written source. In a different thread, there was a vigorous debate about the (non-)use of diacritical marks in English, and it seems that the under-use of diacritics is partially to blame here.
Wikipedia and Wiktionary both claim the etymology of sic is from sīc, "thus, so". Merriam-Webster agrees that it comes from a Latin word for "thus, so" without naming the word. This suggests that the modern word sic is derived from an older word sīc, and could therefore have a slightly different meaning than the ancestral word.
More evidence that the modern sic is not the same sic in the phrase sic erat scriptum is that we write sic and not S.E.S. Compare to Q.E.D. "Q.E.D. is an initialism of the Latin phrase quod erat demonstrandum." (Wikipedia) It seems that when we want to borrow a phrase, we are willing to take the entire phrase in some form, such as Q.E.D. or per se or ad hoc. In this instance we do not write sic erat scriptum or S.E.S., so that allows for the possibility that the modern sic is not identical to the sic in sic erat scriptum.
Finally, the strongest evidence that the modern usage of sic is not identical to the phrase sic erat scriptum is that modern dictionaries describe a usage that slightly diverges from the literal meaning of sic erat scriptum.
insertion in printed quotation to call attention to error in the
original; Latin, literally "so, thus, in this way." . . . Used
regularly in English articles from 1876.
--Online Etymology Dictionary
Note that it reads, "error in the original," but not original text, which allows for original sources that are not text, such as audio recordings.
sic (3), adverb
intentionally so written —used after a printed word or passage to
indicate that it is intended exactly as printed or to indicate that it
exactly reproduces an original
First Known Use: circa 1859
Again, it does not require the original source to be written.
Conclusion: the ancestral usage of sic literally referred to a written source, but the modern English usage, starting around 1855, signals that the quoted source is faithfully reproduced without an error in transcription. Yes, if you do not correct the error from the original source, you may (should?) use sic when writing quotes from aural sources.