People who use racist these days to mean nothing more than prejudiced are diluting the brand. According to Oxford Dictionaries Online, a racist is:
A person who shows or feels discrimination or prejudice against people of other races, or who believes that a particular race is superior to another
So racism is about race. Period.
There are various, more specific words for what you are talking about. Per the eponymous Wikipedia entry on handicapism:
Handicapism is a sociological term, which describes a theory and a set of practices that promote unequal and unfair treatment, discrimination and oppression of peoples with apparent or assumed disability, a concept like to racism or sexism.
But then also there is this, again from Wikipedia’s eponymous entry on ableism:
Ableism or ablism (/ˈeɪblɪzəm/) is a form of discrimination or social prejudice against people with disabilities. It may also be referred to as disability discrimination, ablecentrism, physicalism, handicapism, and disability oppression. It includes apotemnophobia and dysmorphophobia. It is also sometimes known as disablism, although there is some dispute as to whether ableism and disablism are synonymous, and some people within disability rights circles find the latter term’s use inaccurate. Discrimination faced by those who have or are perceived to have a mental disorder is sometimes called mentalism rather than ableism.
But if you really want to get somebody’s dander up, you could call them a crippist: someone displaying prejudice against the crippled.