"As I was saying to Alice, it's been cold recently."
Means "In the same way that I was telling Alice (in an inferred prior conversation), I am now telling you that it has been cold recently."
The person is not only telling you about the cold weather, they are also telling you about their conversation with Alice. So you can infer:
- The person had a conversation with Alice recently
- The person thinks it's been cold recently
- Alice now also knows that the person thinks it's been cold recently
- There's a strong chance that Alice agrees (unless followed up with "but Alice thinks it's been unseasonably mild").
That's a lot of information, that the speaker believes might be of interest to you.
You might reasonably respond:
"Alice? How is Alice these days?"
... which of course wouldn't have been a valid direction for the conversation if they hadn't mentioned their conversation with Alice.
Of course, there are all kinds of reasons the person might want to tell you that they'd been talking to Alice. Perhaps she is a mutual friend; perhaps the person simply wants to name-drop:
"As I was saying to Brad Pitt at the Oscars after-party ..."
You mention: "If person B weren't within earshot they wouldn't say it." -- I don't believe that is the case. The phrase is often used when Alice isn't present.
If Alice were in earshot, then the mention of the previous conversation with Alice would be:
- an invitation for her to join your conversation, and perhaps expand upon what was said
- a recognition that what you are saying isn't a brand new idea; it reassures Alice that the previous conversation hasn't been forgotten.
- in an ongoing conversation, an apology to Alice for retreading old ground, and a hint to you that you shouldn't go into it in too much depth because it may bore Alice.