I propose that your difficulty with this word is not because of the prefix "anti" but because of the suffix "ism".
There are hundreds and hundreds of words in English that end with "ism". (3824 according to the free dictionary.) Here's a few of them: minimalism, classicism, capitalism, literalism, polytheism, etc., etc.
If you view the Wikipedia page on the suffix -ism, you'll find that words with this suffix are often used to describe philosophies, theories, religions, and social movements. So while one person may have a semitic view of things, another may have an antisemitic view, hence semitism and antisemitism. This is how the suffix is used in the vast majority of cases.
The words ageism, sexism, and racism (and perhaps a few others) are really the exceptions to the rule here. For example, racism is not a philosophy or a theology at all. Rather it is used to describe a process of discrimination. It was in the 1920's when the word "racism" really took hold and in the 1960's and 70's other words to describe discrimination were invented that borrowed the morphology.
It makes sense to precede any of the words minimalism, classicism, capitalism, literalism, polytheism, etc. with anti, but not the words that describe discrimination.
So, antisemitism then, is a word of discrimination, not because it ends in ism, but because by definition an antisemite is one who is against the beliefs of Jews. Notice also, that we don't say the person is an antisemitist as we would refer to a person practicing racism as a racist.
To be clear, it is the word antisemite (no ism suffix) that makes us understand discrimination in this sense, not semitism or antisemitism.