I know the meaning but does this word actually exist in English? Should I use it in a formal paper?
Narcotraffic is obviously a compound word. Taking the prefix narco- which is greek for "numbness", but it's clearly, from its common usage, a reference to narcotics. Traffic is form Italian (trafficare), meaning "carry on trade", and may trace back to Latin, but the relationship in meaning becomes less obvious at that point.
(To be pedantic about it, if we assume traffic is Latin, it's a hybrid word, because it combines a greek prefix with a latin root.)
Although there is no universally agreed-upon guideline regarding the use of compound words in the English language, in recent decades written English has displayed a noticeable trend towards increased use of compounds. (from compound)
You should consider it safe for a formal paper if you have found it to be widely used already. You can find several publications (ref) that already use either narcotraffic or narco-traffic. (To answer whether the hypen should be used, see here.)