Well, I will take an intellectual stab at this:
I would doubt it has enough currency outside of tavern signs to be offensive on the face of it.
Calling all members of any group by a word that inaccurately or only describes a small portion of them is typically considered offensive. (Not straight-razors in the parking lot offensive, but still mildly insulting.)
For example, it would be offensive to Pakistani Muslims to be called Arabs on the basis of their religion. It would be offensive to Turks and Persians to be called Arabs on the basis of their religion and Middle Eastern heritage.
I think when it comes to using names for groups, it makes sense to either use their own preferred name or at least that which has been widely accepted.
I do think that the usage is perfectly acceptable in historical context, however. If you are writing a book on medieval Europe, it would be inane not to use the term that was widely accepted for the time. It might be worth noting in a footnote the origin and disambiguation of the term for clarity, but otherwise let fly with all the Saracens you like. I won't tell.