The early history of the term "sexual perversion" indicates that it emerged as a scientific way of describing sexual inclinations that people (including scientists) of the time considered to fall somewhere in an area that ranged from utterly insane to mentally diseased to reprehensible to unutterably abominable.
In the absence of any concerted effort by the scientific community or by people deemed now or in the past to be "sexual perverts" to reclaim the term in a positive or neutral light, it remains weighed down by the old and intensely hostile associations that it possessed or acquired when it first came into use and again when it spread to a broader, nonscientific bloc of users.
The earliest Google books match for "sexual perversion" is from George Beard, Sexual Neurasthenia {Nervous Exhaustion} (1884):
The general term, "sexual perversion," of which Dr. Spitzka speaks in an article on Lord Cornbury, may be used to cover a number of abnormal mental conditions connected with the genital system ; but I see no need, practically, in describing any of these cases, to use any other term than this one, "sexual perversion."
...
Under this head of sexual perversion there are, however, two necessary subdivisions: First: those who are insane, who have the insane delusion—i.e., the delusion that cannot be corrected by the direct evidence of the senses, the delusion that they are women, and who correspondingly assume the manners, the dress, and the customs of women so far as they are able to do. This is simply a monomania, a positive insanity, and of a serious and incurable kind; and it is quite different, essentially, radically,from the following class of cases:
Secondly: those, like the Scythians and the Mujerados and the cases described by Ulrichs, whose sexual instincts are perverted, but who understand that perversion perfectly; who are not under the influence of any delusion, and who are not, in any true sense of the word, insane. This latter class—those who are not insane, but yet have a sexual perversion as a disease, without any delusion, and without sufficient impairment of will-power to make the diagnosis of insanity possible—may be divided into two classes: first those who inherit this tendency or who come into possession of it as soon as the sexual passion appears, or before; secondly, those who acquire this condition as one of the symptoms of sexual debility. In both classes there may be very many symptoms of a nervous impairment.
Basically, Beard divides the universe of people whose behavior qualifies as "sexual perversion" into three classes: the insane; the sane but diseased who acquired their disease by inheritance or very early possession of the tendencies in question; and the sane but diseased who acquired their disease as a symptom of sexual debility. I think it's fair to argue that all three of those diagnoses have negative connotations.
More-specific accounts of types of "sexual perversion" appear in the Medical and Surgical Reporter (September 7, 1889), in the Pacific Record of Medicine and Surgery (May 15, 1890), in The Alienist and Neurologist (1891), and in multiple at least slightly different accounts published in 1892.
Two articles in the 1892 contingent are notable for their preliminary expressions of repugnance about the subject, as a kind of disclaimer, I suppose. G. Frank Lydston, in an editorial published in Medical Mirror (March 1, 1892) begins with this preamble:
The subject of sexual perversion is one which has been studiously neglected by practitioners of medicine, chiefly because it has been considered by them to be an unsavory topic which should be relegated to the realm of moral monstrosities rather than to scientific medicine.
Every bit as eager to deny any prurient motive in its coverage is Journal of the American Medical Association (April 2, 1892):
SEXUAL PERVERSION.
This subject, naturally revolting, has been neglected by the profession, to a very great degree, but the increase in the number of crimes, directly traceable to its influence, which the public press is called upon to record, makes some attention to it almost imperative.
Given the pejorative cast of the original discussions of "sexual perversion," a significant effort to reframe the term as neutral or positive would be necessary to overcome the negative senses surrounding its origin. As far as I know, no such reclamation effort has ever been undertaken—and consequently, the bad old connotations of the term remain strongly associated with it to this day.