Why does Semitic refer to several groups of people, including Babylonians, Assyrians, Arabs and Jews, whereas anti-Semitic only refers to Jews?
NOAD defines anti-Semitism thus:
hostility or prejudice against Jews
And here is the relevant entry from the Online Etymology Dictionary:
also antisemitism, 1881, from Ger. Antisemitismus, first used by Wilhelm Marr (1819–1904) German radical, nationalist and race-agitator, who founded the Antisemiten-Liga in 1879; see anti- + Semite. Not etymologically restricted to anti-Jewish theories, actions, or policies, but almost always used in this sense. Those who object to the inaccuracy of the term might try H. Adler's Judaeophobia (1882). Anti-Semitic (also antisemitic) and anti-Semite (also antisemite) also are from 1881, like anti-Semitism they appear first in English in an article in the "Athenaeum" of Sept. 31, in reference to German literature.
Anti-Semitic is Jewish-specific for historical reasons, as revealed by the Etymology Dictionary. In its most literal sense, anti-Semitic should relate to all Semitic cultures, but this is not the case—a great example of how history, politics, etc, shape English usage.