Basically, when someone says something along the lines of
"I know he's not the smartest person in the world, but he should at least pass first grade math."
It's peculiar because they are using a form of hyperbole to say they are not the smartest person in the world, but what they are really implying is that they are not smart at all.
Edit This post was flagged a possible duplicate, I just wanted to clarify that what I am asking is not the nuance or meaning of the phrase, but the term for the idiom itself. I do not believe this specific question has been asked. Thanks.