We do have a term for that: literal-minded, "tending to take words and statements in their literal sense". In earlier times, however, that phrase meant "unimaginative", though I have not seen it used that way for decades.
- His editing precision was legendary and he was so literal-minded that he even corrected literary quotations.
- They were certainly Gulliver's heroes, but were they Swift's? George Orwell wrote that when he read Gulliver's Travels as a child, he was so literal-minded as to think so.
Edited to add, the noun form is literal-mindedness (with thanks to PLL).
Stanching An Epidemic Of Literal-mindedness
The above is the title of a news article examining why our government — many of whom are lawyers, who use language differently than the average citizen — does not communicate well with the populace. Remember Bill Clinton and his infamous answer: it depends upon what the meaning of is is.