In this question the issue came up as to whether there's any difference in the level of politeness/correctness involved in I'd rather not say as opposed to I'd prefer not to say.
My own gut feeling is the prefer form is a bit more "formal" (and thus arguably more correct/polite). This NGram shows that I'd rather occurs far more often than I'd prefer. Partly that's just because the rather form is more common, but the difference is far less marked with the more formal I would rather/prefer. I take this to mean rather is more suited to casual use/conversation when used with this meaning.
It could be lots of verbs besides say, and I don't think the not is necessarily relevant here either. But usages vary for certain alternatives - for example, go blind only seems to work with rather.
That's just my opinion. Does anyone have arguments/evidence/sources to either back up or refute the proposition that I'd prefer not to say is more formal than I'd rather not say?