As described in the answers to When to use 'an' and when to use 'a' with words begining with 'h'?, some words spelled with "H" start with a vowel sound, and are preceded by an, while others start with a consonant sound /h/, and are preceded by a (with a possible exception for certain speakers when the /h/ is at the start of an unstressed syllable).
It seems that at least part of the variation between a hors d'oeuvre(s) and an hors d'oeuvre(s) that is described in JJJ's answer corresponds to variation between pronunciations with /h/ and pronunciations without it. Although I cannot find a modern dictionary that records a pronunciation of hors d'oeuvre(s) starting with /h/, it seems that such pronunciations exist (or have existed), and may have been more common in the past.
The issue of "an hors d'oeuvre" vs. "a hors d'oeuvre" was discussed a little over a century ago in Notes and Queries. The fact that the H is "aspirated" in French seems to have been thought relevant by some English speakers. (So-called "aspirated" H is not actually pronounced in the textbook standard variety of modern French, but it was lost more recently than other instances of H in the language, so it behaves "as if" it were a consonant for certain rules of standard French pronunciation: "liaison" and "élision". I have also read that "aspirated" H actually survived fairly late as a sound of its own in the pronunciation of some French-speaking regions, as a "dialectal" feature, but this is a complicated topic.)
"Hors d'œuvre."—When passing a book through the press lately, I found my printer correcting my "an hors d'œuvre" into "a hors d'œuvre." This set me making inquiries as to the custom of others in the pronunciation of the word. One friend told me that he sounded the aspirate, because the French aspirated it. Another, a professed French scholar, denied that this was the French habit, and therefore made the h silent in English. Funk & Wagnalls's 1905 dictionary recognizes no option, and omits the h (as also in hors de combat!). The 'N.E.D.' also recognizes no option of pronunciation, and sounds the h (as also in hors de combat). The examples it quotes, however, seem to contradict this, for while two hundred years ago Walpole wrote "a hors d'œuvre," the last example from The Pall Mall Gazette has "an hors d'œuvre." A third friend, older than the others, pooh-poohed contemptuously the possibility of any one ever sounding the h. I next tried Littré. He writes that many have said the h was silent in French, but that it is not so. Finally I asked a lady and gentleman, husband and wife, both English on the father's side and French on the mother's. Both sounded the h in French and in English. Is it possible to ascertain what is the most widely accepted pronunciation of the word in English to-day ?
(Notes and Queries, Tenth Series, Volume X, July–December, 1908. pp. 229–230)
"Hors d'œuvre" (10 S. x. 229). The words beginning with the letter h that, unlike most such, are supposed by French theory to be "aspirated," are not pronounced as Americans, or educated English people, "aspirate" English words which begin with h. Therefore the dispute among Mr. Nicklin's friends cannot be ended by either "Yes" or "No." The French aspirate is a shibboleth, and an Englishman can no more sound a French aspirated h correctly than he can read aloud the common sign of French fur-shops—"Au Tigre Royal," or say "La Tour d'Auvergne."
I think the tendency of an educated Englishman nowadays is to aspirate an initial h. He would astonish his forefathers if they could hear him do it in "herb," in "humble," and in "humour." Hors-d'œuvre is almost naturalized among us, if not quite so, inasmuch as it finds a place in the 'N.E.D.,' and I do not doubt that the editors of that work were right when they insisted on the h being sounded. A useful friend of mine, the 'Nouveau. Dictionnaire,' by E. Clifton and McLaughlin (Paris, Gamier Frères), prescribes the aspiration in hors-d'œuvre, as also in hors, when an adverb unattached. This seems a necessary effort to distinguish the word from or the conjunction, even if or the noun could take care of itself.
(Notes and Queries, Tenth Series, Volume X, July–December, 1908. p. 255)
The authority of the 'New English Dictionary' notwithstanding, I venture to think that "an hors d'œuvre" is better than "a hors d'œuvre."
(Notes and Queries, Tenth Series, Volume XI, January–June, 1909. p. 337)
I'm not sure what's going on with the Google Ngram chart past 2000. The Corpus of Contemporary American English (COCA), which covers 20 million words for each year from 1990-2017, only shows one match for "a hors d'oeuvre" and none for "a hors d'oeuvres" vs. thirty for "an hors d'oeuvre" and nine for "an hors d'oeuvres".