Why does Semitic refer to several groups of people, including Babylonians, Assyrians, Arabs and Jews, whereas anti-Semitic only refers to Jews?
NOAD defines anti-Semitism thus:
hostility or prejudice against Jews
And here is the relevant entry from the Online Etymology Dictionary:
also antisemitism, 1881, from Ger. Antisemitismus, first used by Wilhelm Marr (1819–1904) German radical, nationalist and race-agitator, who founded the Antisemiten-Liga in 1879; see anti- + Semite. Not etymologically restricted to anti-Jewish theories, actions, or policies, but almost always used in this sense. Those who object to the inaccuracy of the term might try H. Adler's Judaeophobia (1882). Anti-Semitic (also antisemitic) and anti-Semite (also antisemite) also are from 1881, like anti-Semitism they appear first in English in an article in the "Athenaeum" of Sept. 31, in reference to German literature.
Anti-Semitic is Jewish-specific for historical reasons, as revealed by the Etymology Dictionary. In its most literal sense, anti-Semitic should relate to all Semitic cultures, but this is not the case—a great example of how history, politics, etc, shape English usage.
Because when the word was coined and came into use, Jews were the only Semitic people encountered in modern European (and American) society.