It doesn't sound right, but I might be mistaken. A quick search shows that people do use it, but they could be wrong as well (I suspect the phrase is ok, and I am wrong).
It's a perfectly valid phrase.
However, it can be said by people who really mean they "refute the idea"; that they prove it to be invalid or untrue, or at least offer a strong argument.
However again, it more often happens the other way around; people say they "refute the idea" either as a hypercorrection (they somehow gather that refuting is "posher" than refusing) or because they fancy they have indeed refuted it, when they've offered no real argument against it, never mind a decisive one, and have merely refused it.
Perhaps you are half-remembering "refute the idea" and that is what has you unsure. All refutations are refusals, but refusals are only refutals if they at least offer an argument for the refusal.