To scream/squall like a mashed cat is a pretty rare turn of phrase, but it accounts for most of the 75 instances of "mashed cat" in Google Books.
Of those 75, the only one I see making any reference to said cat being "flat" is OP's citation. Almost all the others refer to it being angry/howling. Perhaps Carhart chose "cat" because he'd heard "mashed cat" before, in that different context.
It makes little difference whether Carhart knew his usage was "non-standard" for the term - he's a creative writer. As @Jim says, "flat" here primarily means off-key (singing at too low a pitch), but with the obvious allusion to a "squashed" cat being physically flat (as a pancake).
Possibly he intended an oblique comparison between an off-key singer and a screaming/yowling "mashed cat". It's also possible he's alluding to "cats" as jazz musicians from a generation or two ago, and to "mashed" in the modern sense of stoned, intoxicated (a musician in that state might well play flat). But we're getting into realms of Lit. Crit. there, beyond the scope of ELU.