I believe this quote1 can be paraphrased as
It's the little things that wear you down.
Somewhat to my surprise, this phrase (and the similar It's the little things that trip you up) doesn't appear in common lists of idioms, but the set phrase is fairly common, and I find it in Google Books back to 1971.
A typical usage would be
It needn’t be a catastrophe that knocks you off your feet while you’re
chasing your expat dream – often it’s the little things that wear you
down gradually. (Expat Focus online article)
Or this blog post, titled "It's the Little Things that Wear You Down", about the large impact of getting rid of small household irritants.
This phrase is sometimes associated directly with the quote you reference (and Ali):
1It's probably worth noting that Muhammad Ali did not actually coin the original line; it has been around for at least a hundred years, although the original irritant in the shoe was a grain of sand. See Quote Investigator for more on the quote's origins.
Edited to add that "It's the little things..." can have many different endings, e.g. It's the little things that get you down, It's the little things that really get me, It's the little things that will drive you crazy, etc. It's sometimes also used as an exhortation: Don't let the little things get to you and so forth.