The Portuguese explorers or slave traders used the term kaffir to refer to African tribes (the term nigger for 'slave' was also of Portuguese origin); the root word comes from the Arabic for 'non-believer'.
In South Africa, it was only really adopted as a derogatory term by soldiers returning from Egypt after the Second World War had ended. The British had introduced a type of caste system, with the British or English being superior, the Boers providing semi-skilled labour, and the kaffirs or blacks being unskilled and landless cheap labourers. South Africa was a British Union until 1961, by which time most of the legislation depriving black people of the right to own land had already been enacted
(Land Acts 1910-1913).
The Boers have been unfairly blamed for creating this term; animosity between the races was fueled during the Boer War
(a.k.a. the Anglo-Boer War),
where the British military armed black groups to attack Boer farms.
So the answer to your question is quite complex: the word has had many derogatory connotations, and has been used in South Africa for many years. The Bantu Wars, a.k.a. the Kaffir Wars,
took the form of a long-running skirmish; often, the Boers would side with one group of Xhosas against another Xhosa chief, so it was not strictly a war between Boers and Xhosas. The British again got involved. They drove the Xhosas from their land and taxed the Boer farmers, which spurred the latter to embark on the Great Trek (Groot Trek).
In 1976, the term kaffir was banned as hurtful speech in South Africa.