The Ngram chart for "in contrast to" (blue line) versus "in contrast with" (red line) over the years 1750–2005 is quite striking:
After an early period (roughly from 1800 to 1910) when "in contrast with" was a bit more common than "in contrast to," the latter took off and—at the height of its popularity around 1980—was more than four times as popular in published writing as the former. To judge from the trajectory of the two curves, it probably remains considerably more common in such writing today.
But clearly both "in contrast to" and "in contrast with" are widely used, so the next question is whether they mean different things. Theodore Bernstein, The Careful Writer (1965) thinks they do:
CONTRAST Takes prepositions to (opposite); with (different).
In other words, if you are pointing to something that suggests opposition (sharp contrast), Bernstein thinks you should use "contrast to"; but if you are pointing to something that suggests merely some difference (mild contrast), he thinks you should use "contrast with."
Bernstein was a good writer and a smart guy; he was a professor of journalism at Columbia for 26 years, and he served as the assistant managing editor of the New York Times for many years. So he's not just some schmo (like me, for example). But his advice on "contrast with" and "contrast to" falls into the category of aspirational usage: he wants there to be a difference in the way people use the two phrases, but in practice any such distinction is far more often absent than intended.
Merriam-Webster's Dictionary of English Usage (1994) presents a much more realistic picture of actual usage in the second part of its entry for contrast:
2. The commentators are of several minds on the question of the verb contrast used with with or to. Alford [A Plea for the Queen's English] 1866 allows both but prefers to; Bernstein [The Careful Writer] 1965 and Safire [I Stand Corrected] 1984 allow both but think to implies a stronger contrast than with; Longman [Longman Dictionary of the English Language] 1984 and Simon [Paradigms Lost: Reflections on Literacy and Its Decline] 1980 think with the only choice in modern use. The OED finds with more common than to, and with predominates in our most recent evidence.
We find little evidence of a difference of intensity between with and to. With is simply more often used.
I thought perhaps that the Ngram results would be different for "contrast to" (blue line) versus "contrast with" (red line) than for "in contrast to" versus "in contrast with"—but the results are generally quite similar:
The main difference between this chart and the earlier one is that "contrast to" has always been at least slightly more common than "contrast with" in published writing, whereas "in contrast with" had approximately a century of slightly more frequent use than "in contrast to," starting in 1800. On the strength of the Ngram/Google Books evidence, I am baffled by the OED/Merriam-Webster view that "contrast with" is more commonly used than "contrast to."
In any event, you are free to use "in contrast to" or "in contrast with" as you choose; and you can support or ignore the Bernstein/Safire distinction regarding the relative intensity of the two phrases as you like.
My remark above, "On the strength of the Ngram/Google Books evidence, I am baffled by the OED/Merriam-Webster view that 'contrast with' is more commonly used than 'contrast to,'" reflects my underestimation of the effect of instances of "in contrast to" versus "in contrast with" on the Ngram chart for "contrast to" versus "contrast with." As sumelic suggests in a comment below, instances of "contrast with" and "contrast to" in which contrast acts as a verb may be overwhelmed by instances of "contrast with" and "contrast to" in which contrast functions as a noun, in which case the preference for "contrast with" over "contrast to" as verb forms may be thoroughly masked.
This is the point that Jacinto investigated, as noted in his comment below. He created an Ngram chart for "contrasts to" (blue line) versus "contrasts with" (red line), in order to remove the distorting effects of "in contrast to"/"in contrast with" on the verb form frequencies, and the chart looks like this for the period 1600–2005:
As a second test of this overshadowing hypothesis, I created an Ngram chart for "they contrast to" (blue line) versus "they contrast with" (red line), with this result:
Both charts strongly confirm the hypothesis that "contrast with" as a verb is much more common that "contrast to" as a verb, even though for the past century "in contrast to" has been far more common than "in contrast with." My thanks to sumelic and Jacinto for questioning the erroneous assumption in my previous conclusion about the verb forms "contrast with" an "contrast to."