The words are sometimes synonymous. Let's look at other words that are associated with annoy and irritate in dictionaries.
I have taken the key words from Cambridge, Merriam Webster and Collins. Rather than bore you with a list of tedious definitions, this is what I find in my Venn diagram:
The synonymous area is central and orange. Irritate in the pink area is additionally associated with soreness (physical, but could be mental by analogy). Annoy is additionally associated with the words in the yellow area on the right.
I suggest that a focus on the central area produces reasonable claims of synonymy but that there are also differences of meaning shown by the pink and yellow areas. These differences may be brought out in various contexts.