This question is mostly boiled down to opinion or maturity level of the listener.
The word retard has many other uses where it is never considered to be offensive.
For example: fire retardant
Unfortunately, this use has been tainted by the over-usage of the noun retard to describe people who are not mentally retarded. This usage is always always meant to be offensive since it's a comparison of the subject to someone who is mentally retarded.
Mental retardation is still a commonly used medical terminology to describe a symptom of many different syndromes. Due to the negative association, most medical professionals have phased it out in favor of developmental delay and intellectual disability, or, where appropriate, descriptions of the syndrome itself (e.g. Down Syndrome).
There are other words with similar issues. As FumbleFingers points out in comments above niggardly is a prime example. Its similar sound to the offensive n-word makes it a hard word to use in conversation.
So, in direct answer to your question: Use the word judiciously. Consider your audience. If using it to describe a process, Retards the progress of .... it's fine. But, I wouldn't say: We were retarded on our journey.... It's just too fraught with issues.
Note in response to comments: this seems to be primarily an American English phenomenon.