If the question is obviously rude and inappropriate, a blunt, "That's none of your business" would be appropriate. But you asked for a polite response, and even when people are rude to me, I always try to at least start out being polite.
A friend of mine who used to be a political lobbyist once said that when reporters asked her a question that she didn't want to answer, she simply answered the question that she wished they had asked instead. Like:
"Did you make a deal with Senator Jones?"
"This bill will benefit all Americans by reducing unemployment and spurring economic growth ..."
Depending on how adept you are at doing this, it may be obvious that you are avoiding the question or if you are good at it the person may not realize that you have not actually answered his question. In the latter case, problem solved. In the former case, he should get the message that you don't want to answer without you having to be blunt about it. If he doesn't care and presses the question, then I think he quickly forfeits his right to a polite answer. At that point I'd probably just make an excuse and end the conversation. Like:
"Are you and Sally having an affair?"
"Sally really has proven to be an excellent accountant, hasn't she? She's a real asset to this company."
"Yes, but I'm wondering if you two are sleeping together."
"Well, I have to get back to work. Let's chat more tomorrow." Walk away.
If the person actually follows you and demands, "Look, I really want to know about you and Sally."
"If I was having an affair with Sally, it would be none of your business. Excuse me."