According to a number of historians whose books appear in Google Books searches for Saracen + pejorative, the term Saracen was indeed a pejorative term back in the (medieval) days when English Christians widely used it. Hunt Janin & Ursula Carlson, Mercenaries in Medieval and Renaissance Europe (2013) offers this brief discussion:
"Saracen" is a loosely-defined historical term for a group of Arab Muslims, and its meaning has shifted over time. Initially, in early Greek and Latin, it referred, apparently non-pejoratively, to the peoples living in the desert areas in or near what the Romans called "Arabia." By the Early Middle Ages, however, this term was being used in a more negative sense to describe all the Arab tribes.
From Frederick Quinn, The Sum of All Heresies: The Image of Islam in Western Thought (2008):
The origin of the word "Saracen" is uncertain; by the Christian Middle Ages it had become a sweepingly pejorative term applied to almost all Arabic-speaking populations. "Saracens" was also a play on the Greek word sarakeonoi, meaning "easterners." The word Islam was unknown in these centuries and did not appear in English until the seventeenth century in works like Pilgrimage (1613), an early survey of peoples of the world and their religions by Samuel Purchas (1577?–1626), a chaplain to the archbishop of Canterbury and compiler of early travel accounts.
From Debra Strickland, "Monstrosity and Race in the Late Middle Ages," in The Ashgate Research Companion to Monsters and the Monstrous (2013), which draws a parallel between Saracen and Tartar in medieval usage:
It is an equally short conceptual step from walking, talking, well-dressed (if cannibalistic) Dogheads to medieval Christian characterizations of Muslims, known pejoratively as "Saracens," and characterized by some Christian chroniclers as "a race of dogs." How this thought pattern might have colored perceptions of the Mandeville author's polygamous and idol-worshiping "Tartars," the pejorative medieval designation for Mongols, is suggested in one English copy of this work, ...
From Jerold Frakes, Contextualizing the Muslim Other in Medieval Christian Discourse (2011):
If "Saracen" does not mean "Muslim," does it mean "Arab"? Indeed, when it is first used by Greek and Latin authors of late antiquity, it is used principally to designate nomadic Arab tribes living along the southern and eastern borders of the Roman Empire. The term was not, originally, used to refer to the settled Arabs living within the empire. Yet over the centuries, the term came to designate Muslims (including non-Arab Muslims), though it never completely lost its ethnic overtones. Arab Christians are never called Saracens. After all, according to Jerome, the Saracens were the descendants of Ishmael, son of Abraham and his slave Hagar: if they call themselves "Saracens," it is because they try to usurp the heritage of Sarah, Abraham's legitimate wife. This pejorative etymology was echoed by countless medieval authors, though of course it ignored the fact that the "Saracens" in fact did not call themselves "Saracens."
And finally the entry for Saracen in Michelle Sauer, The Facts on File Companion to British Poetry Before 1600 (2008), which brings us up to the "early modern" period:
SARACEN The word Saracen is an English adaptation of the Greek word sarakenos (easterner). It was used commonly in medieval and early modern British literature to refer to any non-Christian, non-Jewish person, usually from the Middle East but also possibly from North Africa or even Spain; Arab or Muslim are rough synonyms. The use of the term is usually pejorative and indicates an opponent of Christianity. It is seldom attached to actual cultural knowledge; instead, most literary depictions of Saracens involve simple behavioral stereotypes (treachery, greed,cowardice), either for comic effect or as part pro-Christian propaganda.
The pejorative sense persists into the late eighteenth century—in a comedic setting, at least—as we see in these lines from Thomas Holcroft, A Knave or Not?, second edition (1798):
Sir Job Ferment [to his son]: Do you forget that I have a right to knock you down when I please? Bombs and gunpowder, get out of my sight. (Aside) You may follow me——Begone, you vile—— what shall I call you?——See to the settling of that account with Paywell and Co. Five hundred nineteen seven and six in our favour. Get about your business you, you, you Saracen Turk!
But Karim H. Karim, Islamic Peril: Media and Global Violence (2000) [combined snippets] uses Saracen in its old terror-fraught sense in describing modern Western views of Islam:
Violence committed by militant Muslims is usually placed within journalistic frameworks whose cultural roots are hundreds of years old. For example, editorial cartoons draw on images such as the bloodthirsty Saracen wielding "the sword of Islam" embedded in medieval European literature. Such depictions tend to hinder the understanding of violence as well as of Islam.
So what we have with Saracen is a term that for centuries was used in a highly pejorative way to demonize an ill-defined category of people centered on Muslim Arabs, but that gradually fell into disuse as English speakers adopted more-precise terms to differentiate among members of the broader class formerly labeled "Saracens." I get the impression that the main survival of Saracen in the UK is in the form of references to pubs and inns styled The Saracen's Head that dot English literature of the eighteenth and nineteenth century; the wording seems rather quaint in that setting. In the United States, Saracen is hardly known or used at all.
But any attempt to dust the word off and use it today in a neutral sense must deal with (or overlook) the fact that it served in the West as an exceedingly hostile epithet that, ethnologically speaking, was very poorly defined anyway. I can't think of any good reason to try to rehabilitate it, given that its meaning was so negative for so long, and so vague in the first place.