Oppositional (adj) came to mind. Also the phrase arguing for the sake of arguing.
And some of those people might call themselves a devil's advocate. As this link shows, however, a devil's advocate takes a position he or she doesn't necessarily agree with for the sake of argument (not for the sake of ruffling feathers). The purpose of playing devil's advocate is to test the validity of the other side's arguments.
I'd also add that while perhaps not done for the same reasons as you have cited, to play devil's advocate requires taking and arguing a contrary position- but with a purpose of evaulating an issue from all sides rather than just being polemical.
A relevant meaning of stickler is "One who is extremely fussy or particular; one who insists on precision or correctness". (An irrelevant meaning of it is "A referee in a Cornish wrestling bout".) Note that the described kind of person may be a stickler, but a stickler is not necessarily trying to ruffle feathers.
Anyhow, I might also refer to such a person as bristly, prickly, spiny, or starchy, or more figuratively (but closer in spirit) as a kerfuffler.