It was La Rochefoucault. Robert Stern explains this as
In an earlier age, La Rochefoucault could still laugh at hypocrisy as "the compliment vice pays virtue", but he was only dealing with the naive hypocrisy of a Tartuffe. Moliere's hero, after all, just pretends to be more pious than other people in order to cover up his wicked schemes. Tartuffe's hypocrisy was merely one form of unctuous fraud among many others.
It's an odd construction, but based on imitation being the sincerest compliment, vice will imitate [try to look like] virtue. It's hypocritical in the case of vice because it's not sincere at all.