It's worth pointing out that the word "Tory" is itself often used by opponents of the Conservative party as an insult. I am a Conservative activist in the UK, and I recently discovered that, in their internal polling system, the Labour Party uses the letter "T" to denote "Conservative". In contrast, in our internal polling system, we use the letter "C" to denote the Conservative Party. Interestingly, we use the letter "S" - for "socialist" - to denote "Labour" in our system (with "L" meaning Lib Dem), as we have historically considered that to be an insult.
It should be pointed out, however, that many Conservative activists - myself included - don't view the word "Tory" as an insulting term, and in fact wear it with pride.
If this seems confusing, it's because the word "Tory" refers to a specific kind of conservative ideology in the UK (and North America) which predates the contemporary Conservative Party. The Conservative Party was formed out of the historical Tory Party by Prime Minister Robert Peel in the 19th Century, and this represented a significant ideological as well as cosmetic change. The "Conservative Party" came to represent a blended ideology which included parts of the ideology of both the historical Tories and the historical Whigs. As time progressed, this fusion of ideology deepened as two things - Irish nationalism and, later, the rise of socialism - led to more and more supporters of the (historical) Liberal Party (which represented views which today would be - roughly - thought of as right-wing libertarian) joining the Conservatives and thus increasing the "Conservative" commitment to the historically "liberal" values of free markets, capitalism, etc - which the Tory Party had always viewed with some suspicion as vulgar and disruptive to the social order.
However, many of us Conservatives still feel a strong sense of affinity with the historical Tory Party and (to some degree) its ideology, which, among other things, ended child labour, was crucial to the abolition of slavery and emphasises British Unionism, patriotism and the empire - particularly affinity with the "Dominions" of Canada/Australia, etc - which is why the Canadian Conservatives are also called "Tories".
Basically (simplifying massively), "Conservative" refers more to the modern, Thatcher/Reagan-style conservatism of free markets, individual liberty and social conservatism in the "family values" American sense. "Tory", on the other hand, refers more to traditional, British conservatism in the sense of the monarchy, slower social evolution, a hierarchical (but benevolent) social structure, the establishment of the Church of England and the Church of Scotland (technically a "national", rather than "established" church) and (historically) support for British imperialism. To some degree, many British Conservative/Tory supporters feel an affinity with both identities. Complicating things further, some modern Conservatives will pointedly correct you if you call them Tories, and tell you that they are proud Whigs!
In "class" terms, historically, the Tory Party also represented more the petty aristocracy and middle-class merchants, whereas the Whigs represented more the elite aristocrats and wealthy industrialists, although this see-sawed a lot throughout history and was never anything like that simple.
This is a very brief summation of a very complex issue, which whole books could be written about. If you are writing fiction, you are probably safe to assume that Conservative supporters will refer to themselves as "Conservatives", while their opponents will refer to them as "Tories" as an insult - but be aware that this is a massive simplification.