The phrase "coming out" has a fairly rich history of figurative usage stretching back to at least 1637. My answer focuses on three senses of term, in order from oldest to youngest: "making an appearance," "entering society," and "publicly avowing one's homosexuality."
'Coming out' as 'making an appearance'
J.S. Farmer & W.E. Henley, Slang & Its Analogues (1891) has this first definition of "come out":
COME OUT, verbal phr (common).—1. To make an appearance; to display oneself; to express oneself vigorously; to make an impression (especially in sense 2 ["To turn out; to result"]) Sometimes in an intensified form TO COME OUT STRONG.
Farmer & Henley then lists two examples, both involving the form "coming out," although the authors warn that "The first quot. is doubtful, but it looks like an anticipation." Here are the fuller versions of those two citations. From Samuel Rutherford, letter to "my Lady Boyd" (May 1, 1637), in Joshua Redivivus, or Mr Rutherford's Letters (1765):
Madam, our debts of obligation to Christ are not small; the freedom of grace and salvation is the wonder of man and angels: but mercy in our Lord scorneth hire; ye are bound to lift Christ on high, who hath given you eyes to discern the devil, now coming out in his whites, and the idolatry and apostasy of the time well washen with fair pretences; but the skin is black, and the water foul: It were art I confess, to wash a black devil and make him white.
And from William Thackeray, The Newcomes: Memoirs of a Most Respectable Family (1855):
Then Clive told us of his deeds during the winter; of the good fun he had had at Rome, and the jolly fellows he had met there. Was he going to astonish the world by some grand pictures? He was not. The more he worked, the more discontented he was with his performances somehow: but J. J. was coming out very strong, J. J. was going to be a stunner.
'Coming out' as 'entering society'
The earliest dictionary notice of "coming out" in the sense of "entering society" that I have found is in John Bartlett, Dictionary of Americanisms, fourth edition (1877), where it appears as the third definition in the entry for "To come out." All three definitions are worth repeating here, however:
To come out. 1. An expression used among certain religious enthusiasts, meaning to make an open profession of religion. [Example:] I experienced religion at one of brother Armstrong's protracted meetin's. Them special efforts is great things,—ever since I come out, I've felt like a new critter. Widow Bedott Papers , p. 108.
2. "How did you come out?" means, how did you fare in your undertaking? Come off would be more agreeable to English usage. To come out at the little end of the horn means to fare badly, to fail. [Example omitted.]
3. A young lady when she first makes her appearance in society is said to come out. [Example:] Clara, just seventeen, and a very pretty girl, is looking forward with impatience to next year, and coming out in society. — Miss Gould, Marjorie's Quest, p. 46.
Bartlett only slightly alters the quotation from Jeanie Gould, Marjorie's Quest (Boston, 1872). Here is the original example:
There were seven of them [children of Mrs. Maxwell]: Percy, two years older than Regie, a Freshman at Yale and very superb accordingly (that is, at home; when at college he endured a sufficiency of snubbing from those intolerable Sophs); then Clara, just seventeen, looking forward with impatience to next year, and "coming out" in society.
This definition of "to come out" is new to the fourth edition of Bartlett; the third edition (1860) lists only the first two definitions.
A Google Books search turns up an even earlier instance of "coming out" in the social sense of the phrase in a novel published in London in 1835—with the difference that the person coming out is male. From James Payn, From Exile (1835):
Mr. Bree always behaved himself; he did not suck his knife, or prefer the use of the back of his hand to that of a table napkin; he did not grunt defiance, nor laugh when he shouldn't in a coarse, demonstrative manner, or stand with his hat on very much on one side (as some people did) in the presence of the ladies of the house. On the contrary, he was very polite; indeed, so studiously so that one would almost think he had refreshed himself after his legal studies, before coming out into society, by the perusal of some work on etiquette.
And soon after Payn, another book published in London took up the female side of coming out. From Eliza Farrar, The Young Lady's Friend; A Manual of Practical Advice and Instruction to Young Females on Their Entering upon the Duties of Life, After Quitting School (1837)
I asked a gentleman once, if he did not think Miss C––––– very pretty and ladylike;—"I used to think so," was his reply, "but I saw her treat an old gentleman rudely, and she has never looked pretty to me since." I often think of this, when I see young girls just coming out into society pushing themselves before their elders, elbowing their way to the supper-table before wives and matrons, accepting the attentions of elderly gentlemen as if they were conferring, instead of receiving favours, and treating those whose age and station entitle them to marked respect from young people, with as much careless familiarity as they would their own classmates.
Mrs. Lincoln Phelps, Hours with My Pupils: Or, Educational Addresses, Etc. (New York, 1859) devotes considerable space in "Address II: Dignity of Character" (originally delivered in 1842) to the subject of coming out:
False notions of life often fill the minds of young girls about to leave school. The expressions "coming out into society," "finished education," "entering life," etc., seem to mean something, though what this something is, cannot a;ways be easily defined.
The "coming out" of a young lady, or her being permitted to partake freely of fashionable dissipation, is but a poor beginning for a life of duty, trials and cares. It would be far better if, for the foolish notions which too often fill the heads of young girls, could be substituted more just and rational ideas of life. ... The idea that in "coming out," a young lady must have admirers, and that the greater their numbers the greater her triumph, naturally leads her into folly and flirtations. In her haste to secure beaux she perhaps loses a worthy and devoted admirer, who becoming disgusted with her frivolity and apparent heartlessness, leaves her for one less brilliant, but more worthy of his affection.
But as interesting as the debutante meaning of "coming out" is, use of the term to describe a religious enthusiast's open profession of faith is even more striking (to me) as an antecedent for the much later gay sense of "coming out." That sense of the phrase appears as early as the second edition of Bartlett (1859), with further details in the associated entry for come-outers, which first appeared in the first edition of Bartlett (1848):
COME-OUTERS. This name has been applied to a considerable number of persons in various parts of the Northern States, principally in New England, who have recently come out of the various religious denominations with which they have been connected; hence the name. They have not themselves assumed any distinctive organization. They have no creed, believing that every one should be left free to hold such opinions on religious subjects as he pleases, without being held accountable for the same to any human authority. ... —Evans's History of Religions, with Additions by an American Editor
The religious sense of "coming out" is absent from Farmer & Henley, however, so it may have dropped out of use (and memory) by the end of the nineteenth century.
'Coming out' as 'publicly avowing one's homosexuality'
J.E. Lighter, Random House Historical Dictionary of American Slang (1994) has this entry for "come out":
come out v. Homosex. to openly acknowledge one's homosexuality, esp. by beginning to engage in homosexual acts. ... [First three cited examples:] 1941 G. Legman, in G. Henry Sex Variants II 1161: Come out To become progressively more and more exclusively homosexual with experience. 1949 Gay Girl's Guide 5: Come Out: To be initiated into the mysteries of homosexuality. 1963 Stearn Grapevine 5: The phrase "coming out" ironically indicated a lesbian's sexual debut: being "brought out" reflected the same result.
The earliest instance that a Google Books search finds of "coming out" used in connection with an open profession of one's homosexuality is in Erving Goffman, "The Moral Career of the Mental Patient" (1958), reprinted in Goffman's Asylums: Essays on the Social Situation of Mental Patients and Other Inmates (1961). Goffman begins by describing the acclimation of recently arrived patients to life at a mental hospital:
Usually the patient comes to give up this taxing effort at anonymity, at not-hereness, and begins to present himself for conventional social interaction to the hospital community. ... Sometimes this making of oneself available is called "settling down" by the attendants. It marks a new stand openly taken and supported by the patient, and resembles the "coming-out" process that occurs in other groupings. 28
Then, in the accompanying footnote 28, Goffman describes one of the "other groupings" he has in mind:
28 A comparable coming out occurs in the homosexual world, when a person finally comes frankly to present himself to a "gay" gathering not as a tourist but as someone who is "available." See Evelyn Hooker, "A Preliminary Analysis of Group Behavior of Homosexuals," Journal of Psychology, XLII (1956), pp. 217–25; see especially p. 221. A good fictionalized treatment may be found in James Baldwin's Giovanni's Room (New York: Dial, 1956), pp. 41-57. ...
"Coming out" has been used figuratively to mean "making an appearance or displaying oneself" for centuries. The example from 1637, cited in Farmer & Henley, Slang & Its Analogues, seems legitimate to me, though the authors of that dictionary say rather that "it looks like an anticipation."
"Coming out" in the sense of "entering society" goes back to 1835 (with reference to a young man in England), to 1837 (with reference to young women in England), and to 1842 (with reference to young women in the United States) in Google Books searches. A now-obsolete use of the phrase—at least in the infinitive form "come out," to mean "make an open profession of religion"—was current by 1848, according to the first edition of Bartlett, Dictionary of Americanisms.
And finally, "coming out" in the sense of "presenting oneself openly (at least in some settings) as homosexual" is documented as far back as 1941 (in Lighter, Random House Historical Dictionary of American Slang) and 1958 (in Google Books search results).
As to the specific question, "Was the 'the debut in society' the origin of its usage [with reference to homosexuality]?" you could certainly make a strong argument that the two are directly connected. But other possibilities exist as well, and at least one other use of "coming out" was gaining steam during the 1930s—the decade prior to the first citation in Lighter: "coming out of [one's] shell."
A Google Books search for "coming out of his shell" finds one match from 1903 (from Bernard Shaw's Man and Superman), another from 1930 (from Gordon Stowell, The History of Button Hill), another from 1931 (from L'École Canadienne, Revue Pedagogique), and another from 1933 (from Richard Hatch, Leave the Salt Earth)—all in the sense of "emerging from self-imposed (and perhaps self-protective) isolation."
One phrase that seems to have emerged too late to take credit as the source of "coming out" is the longer wording "coming out of the closet." The first Google Books match for this phrase is from Douglas Stange, Metanoia, volumes 2–8 (1970[?]):
We are, some of us, coming out of the closet, or at least turning on the lights. We are finding out who one another are, and affirming each other in living a Christian lifestyle that is also gay. And, shoulder to shoulder with our non-gay Lutheran brothers and sisters, we are extending to our church, as individuals and through our thirteen chapters across the country, a ministry of education and understanding.
J.E. Lighter gives a first citation for "come out of the closet" of 1971, and a first citation for "in the closet" and "closet queen"—both from Mart Crowley's play, The Boys in the Band—of 1967.
Historically "coming out" has been used idiomatically to refer to a number of ideas associated with taking one's place openly and visibly in the world. It may be that multiple senses of the phrase influenced the emergence of the phrase in a specifically gay milieu. But because homosexuality was such a taboo in the United States and other English-speaking countries, particular terms may have evolved underground (as it were) over an unknown number of years, and that evolution in the shadows may make tracing the origins of gay terminology unusually difficult.