Wallflower was used of men as early as 1839, just nineteen years after the first known use of the word in reference to women. However throughout the 19th century one finds examples of usage where the author appears to expect the reader to understand by default that wallflower referred to women.
The earliest instance of wallflower I could find in Google Books in reference to men is in Albert Smith in “Sketches of Evening Parties. The Wallflowers,” The Literary World: a Journal of Popular Information and Entertainment, No. 13, June 22, 1839, p. 197 (my emphasis in all quotes):
Sketches of Evening Parties
The Wallflowers
Do not think, gentle reader, we are about to commence a descriptive lecture upon Botany […] we are taking into consideration a more interesting class―a link between the animal and vegetable kingdom―a zoophyte―in a word, the wallflowers of an evening party. […] the real living people that border the walls of the apartment, and seem always to live on terms of the closest intimacy with the rout-stools and ottomans.
If you study their manners and instincts with the enthusiasm of an amateur, you will find amongst them a most striking set of young men, whom you will be sure to meet in shoals at every party you attend. They are very modest and retiring―too much so in the opinion of the dance-loving young ladies, and they are seldom seen in the quadrille; indeed we could never precisely understand what they come for, unless it is to fill up the room and get in the way of the waltzers.
Albert Smith went on to develop a little industry on this subject, and published a few more tracts in book form and in magazines on both sides of the Atlantic incorporating, word-for-word, chunks of the original Sketches. See The Wassail Bowl (1843) and The Physiology of Evening Parties (1846). Here’s another cute passage from “Fashionable Life in New-York. Evening Parties.” The New World, Vo. VI, No. 10, March 11, 1843, p. 288:
The Wallflowers. ―This numerous class is not strictly composed of mesdemoiselles un peu passées, or of short, homely married gentlemen, who have tall, handsome wives, or of young men, who don’t know what to do with their hands, but in it may be found valuable young women―blue stockings, “azure hose,” who have read Liebig’s Animal Chemistry and gone extensively into Transcendentalism, and whom all the dandies are afraid of―because “they are so sarcastic.”
Also in 1843 (google books wrongly indicates 1813) we have from Ann S. Stephens, “The Wife,” Graham’s Magazine, Vol. XXIV, No. 5, Philadelphia, November, 1843, p. 242:
“Hay Burke, are you here playing the wallflower?” said the young guardsman, as he turned from escorting his partner to a sent. “How is it that I have not seen you among the dancers?”
And in 1854 from Francis Alexander Durivage, The Three Brides: Love in a Cottage and Other Tales, The Library of Alexandria, 1854:
“Tell me, Charley, who is that fascinating creature in blue that waltzes so divinely?” asked young Frank Belmont of his friend Charles Hastings, as they stood “playing wallflower” for the moment, at a military ball.
Then in 1872, from Martha Finley, Elsie’s Girlhood, The Library of Alexandria, 1872:
“Since my misfortune compels me to act the part of a wallflower, I am selfish enough, I own, to rejoice in your decision to be one also,” he said gleefully.
Curiously, except for the work of Albert Smith, in all other examples the man only “plays” or “acts the part of” the wallflower; it appears that he is not one himself. Then we have other examples where it looks as though the reader is expected to know that wallflower refers to women before that is made clear, as in this letter to the editor of Vermont Phoenix, February 4, 1847:
The [undercrust’s] merry dance went on―the music was superb―nothing occurred to mar the festivities of the evening till near midnight, when lo!―this being the ‘witching time of night when ghosts do stalk forth’―in marched a whole phalanx of he-uppercrusts, probably to take a peep at the “wall-flowers.” The uppercrusts were enchanted ― dazzled ― bewildered ― mesmerized by such a display of brilliants!―and taking advantage of a cessation in the exhibition of “heel and toes,” stepped forward and plucking each a “wall-flower,” led her out, and forming two and to in regular order, the music struck up […]
In these two examples wallflower is used for women and something else is used for an equivalent male:
We regret that this pretty little volume did not reach us in time for notice previous to the Holidays. It will be found and ornament to that refuge of the ennuyéed, the centre-table , and whoever wishes to have his fortune told, may be sure of a graceful and pithy response to the questions, so that hope of future joys may relieve the frequent martyrdom of evening parties. The wall-flower may thence win a promise that she shall yet become a Rose, and the youth, too green a grape for even foxes’ food, look forward to the time when he shall be the pride of the vintage.
[Review of Sibylline Verses by Miss H. J. Woodman in New-York Daily Tribune, January 6, 1848]
No Wallflower should be left without I partner if she can dance, and no bashful man should be passed over without introducing him to some one who will take pity on awkwardness.
[Frank Leslie’s Lady’s Magazine, Volume 38-39, 1876]
So it looks as though wallflower was used of men very early on, but would still by default be understood to refer to women.