I am having a really hard time seeing the nuanced differences between these three synonyms, especially vituperation and vitriol. I saw from my research that invective is used in more formal context with use of refined language (if somebody can add more to this word, I would be appreciate it). But I have trouble seeing the subtle difference between vitriol and vituperation since both suggest bitter, cruel, harsh abusive language.Thanks.
After consulting a couple of online resources such as the Merriam-Webster and Cambridge dictionaries, it seems to me that at least a number of partial answers is available in popular language reference works.
Here's the Merriam-Webster's (MW) take on the small distinctions between vituperation and some of its synonyms and words that come close to it in meaning and usage.
As for vitriol (which happens to be covered as the last item in the synonym list there) vs. vituperation, it seems to me that it may be not so much a matter of precise meaning or expression on a factual level as of register or approximative position on a formality scale.
Compare, for a similar case illustrating register, the example of some dated subway sign saying
Expectoration is forbidden
to express what could be more plainly put as
Please do not spit
(Note: Source has eluded me but could be retrievable if needed.)
For an attempt at a partial seat-of-the-pants answer (another user may be able to give more insight as to this), vitriol seems to be rather towards the everyday/colloquial where the more formal and neutral term seems to be abuse while vituperation seems to me to have a bit of a formal touch for whatever intent or purpose. (Such as the speaker distinctly wishing to sound distinguished and somewhat above any person whose demeanor they may describe as shedding vitriol.)
Here's an example of the use of vitriol from somewhere in literature (last example on the page):
Two men positioned themselves directly behind me and from them came a constant stream of vitriol.
On a personal note, that lets me think of a street scene of a certainly not elevated tier of society, at least for the two men mentioned. As a matter of taste (YMMV), it seems that here it would be breaking register (in the language of the narration that might credibly be kept in accordance with the content and inner setting of the narration) to use vituperation.
So at least part of the matter does boil down to the subtleties of using words of potentially the exact same meaning (depending on context as there are rarely two words of overall exactly mutually overlapping content) to express things in one's speech other than just the plain factual meaning, such as mood and register.