I found one use of the phrase on a Prince fan site's discussion of the Beastie Boys. This is from an 8/16/11 post from user Cerebus (quoted in full in honor of MCA):
Been a diehard fan since License To Ill, which is like, 25 years ago. Been listening to the Beastie Boys almost as long as I've been listening to Prince. I love them. I love that they've changed over the years but are still clearly the Beastie Boys.
However, while I can still get way into all the older stuff (especially Check Yo Head and Ill Communication), I find it much more difficult to really love the last two albums. I dug what To The 5 Burroughs was about and I enjoyed that they took the beats back to an early electro hip-hop sound, but I still don't think its a great album. The new album, for as long as it took to get it done (regardless of the health issues), didn't really frost my pickle. Its good, and I can feel the magic trying to creep in, but again, not great.
I'll always be a fan and I'll always support them (including going to live shows if they ever tour again), but I think my real deep love for them probably ended with Hello Nasty.
So here it's being used in a positive way.
I found the annoyance sense of the phrase at a Yahoo Answers question from a woman divorcing her husband and looking to sell his baseball card collection. This is the top-voted answer:
well, hell, if's revenge you want why bother trying to sell them, give em away and tell him you did, that ought to frost his pickle!!
Could either of these posters be your mom?
Edit-- Keep finding more:
-One thing which frosted my pickle was no internet,and no
wireless.In 2011 this inexcusable
-I kept getting hassled by guys who thought they were smoooooooooooooth… their primary means of greeting females being to smack them on the butt. This really just frosted my pickle.
-I showed up for the meeting only to discover that it had been rescheduled. Now that really frosted my pickle!
I'd say the annoyance meaning wins out. Still no clue where it comes from, but it's definitely out there.
Edit #2:
After a little more looking around, I've found that the construction:
Well, don't that just frost your [fill in the blank]!
is a fairly common US colloquialism used to express annoyance or, more specifically, sympathy for someone being annoyed or vexed. In addition to pickles, the various things being frosted in these expressions include: flakes (after the cereal), balls (testicles), knuckles, feathers, asses, hides, and cakes.
I kept thinking this referred to frost as in damaged by cold, but this last one may hold a clue. These phrases could be a sarcastic reworking of the idiom
the frosting (icing) on the cake
by humorously emphasizing the unnecessary aspect of what has just happened. However, NOAD also has this informal definition of frost:
anger or annoy
such discrimination frosted her no end
So the question remains, is this informal use of frost a result of these colloquialisms or did frost carry this meaning before these colorful sayings were developed?