In casual conversation, even in written discourse, has English evolved to where using “whom” can slow down readers and confuse listeners ?
I think it has come to that point, at least for a lot of people. As I've been pointing out for decades, there is never any case in English where one must use whom, but there are an infinite number of cases where one must not. Anyone who uses whom takes on the responsibility of using it correctly, which means they are implying that they understand enough of English syntax to use it correctly. This is a very difficult claim to justify.
There is one situation, as @PeterShor mentions in a comment, where it is required, but that situation is both rare and optional, so one need never go there.
- If one forms a relative clause where the relative pronoun refers to a human being, and
- If that relative pronoun is the object of a preposition, and
- If one optionally decides to use a relative pronoun, instead of omitting it, and
- If one optionally decides to use who as the relative pronoun, instead of that, and
- If one optionally applies the rule of Pied-Piping, which moves the preposition with its object,
Then, and only then, is whom required. Exemplorum gratia:
- I threw the ball to Bill.
- Bill is the man I threw the ball to. (relative pronoun deleted)
- Bill is the man that I threw the ball to. (that instead of who, with stranded preposition)
- Bill is the man who I threw the ball to. (who with stranded preposition)
- Bill is the man whom I threw the ball to. (whom with stranded preposition)
- Bill is the man to whom I threw the ball. (whom with Pied-Piped preposition)
In the last sentence above, only whom can be used. In the ones previous, it's optional, and one can also use who, or that, or nothing. So avoid pied-piping and strand those prepositions like a native speaker, and you'll never have to worry about whom again.
The main problem is that so few people know the actual rule, but most people think there must be some grammatical rule, and they're in the usual Anglophone state of "anxious cluelessness" about syntax, as Geoff Pullum calls it, so they make up some rule that makes sense to them, and follow it. These rules often have nothing to do with grammar per se, but rather about "formality", "correct English", and other vague generalities, and the result can be pretty silly at times. Like this, for instance.
And the upshot is that about half the time whom is used incorrectly by the actual rule, so anybody who does know it (or anybody who doesn't know it, but has some other rule instead) is likely to pause and check to see whether the speaker or writer knows what they're doing. Often enough, the conclusion is that they don't. Whether they're right or not, they will be perceived as wrong by some.