I have been told that it's a racist derogatory term.
If someone told you it is a racist derogatory term, they are probably referring to red-haired-individuals.
In some countries like the UK, ginga (pronnounced jin-ja, not gin-ga) is a slang form of "Ginger".
See Urban Dictionary Ginga
Slang U.K Ginger
See Wikipedia Red Hair - Modern-day discrimination
In British English, the words "ginger" or "ginga" are sometimes used to describe red-headed people (and are at times considered insulting)
Some other countries from English speaking history like Australia, has similar slang but it's "Ranga" as in "Orangatang" implying the orange/red hair the animals have.
See Urban Dictionary Ranga
a person who has red or orange hair