I've never heard that used as a derogatory term. But then I've never moved in circles rich enough to hire gardeners. The conventional derogatory term for Mexicans in America is "wetback". But if some are now using this term as an insult, I guess you have to be careful with it.
If a Mexican is a skilled brain surgeon and you called him a "leaf blower", I guess he'd take that as an insult. But if two Mexicans were working on the lawn and you said, "Jorge is the weed puller and Pedro is the leaf blower", that doesn't sound insulting to me, but a simple statement of fact. If someone said he was offended by that, I'd just reword it. Like say, "Jorge is pulling the weeds and Pedro is cleaning up the leaves" or some such.
If someone routinely refers to all Mexicans as "leaf blowers" ... well, it sounds like an odd insult to me, but he presumably intends it as an insult. People can be super-sensitive about words, but if someone is clearly TRYING to insult you, I think you have a right to be offended.