According to a grammar book, it is incorrect to state "The company directors realized too late that the man that they had just brought in was unsuited to do the job." They claim it is better to state: "The company directors realized too late whom they had just brought in was unsuited to do the job". 

Why is using "that" wrong here, when Oxford Dictionaries claims that “that” can refer to objects and humans alike (per [Tragicomic's answer to *How to use “who” vs. “that”*][1])?


  [1]: https://english.stackexchange.com/a/11339/77227