Starting in Early Old English, "man" was used to refer to a human, without respect to sex. After Old English, the usage of the gender-neutral definition was restricted to to refer to a generic humans or humans in general. Starting around 1000 (which is during Old English), "man" was also used refer to a male human, so it would have been used in a purely gender-neutral way only before then. Using "man" to refer to humans without respect to gender started falling out of favor in the late 20th century, due to the influence of feminism.
Specific sense of "adult male of the human race" (distinguished from a woman or boy) is by late Old English (c. 1000);
implying that "man" was used in a purely non-gendered way before then.
Indeed, OED's quotations for the definition of "a human being as a designation applied equally to particular individuals of either sex" are earlier than the ones for the definition referring to male humans. (eOE, Early Old English, is considered to be from 600–950.)
eOE Leechbk. (Royal) (1865) iii. xxxviii. 332 Gif wife to swiþe offlowe sio monað gecynd, genim niwe horses tord, lege on hate gleda, læt reocan swiþe betweoh þa þeoh up under þæt hrægl, þæt se mon swæte swiþe.
In contrast, the earliest quotation for the definition "a male human" is circa 1000:
OE Ælfric Lives of Saints (Julius) (1881) I. 28 He..sæde hyre gewislice hwæt heo man ne wæs.
Old English used wer and wif to distinguish the sexes, but wer began to disappear late 13c. and was replaced by man. Universal sense of the word remains in mankind and manslaughter. Similarly, Latin had homo "human being" and vir "adult male human being," but they merged in Vulgar Latin, with homo extended to both senses.
Man also was in Old English as an indefinite pronoun, "one, people, they." It was used generically for "the human race, mankind" by c. 1200. As a word of familiar address, originally often implying impatience, c.1400; hence probably its use as an interjection of surprise or emphasis, since Middle English but especially popular from early 20c.
Oxford English Dictionary comments:
A human being (irrespective of sex or age).
Man was considered until the 20th cent. to include women by implication, though referring primarily to males.
The last quotation for the non-gendered sense of "man" (and not "men") referring to a specific individual is also from Old English:
Ælfric's Homily De Initio Creaturae (Vesp. A.xxii) in R. Morris Old Eng. Homilies (1868) 1st Ser. 223 He com þa anedren hiwe toðam twam mannum, erest toðan wife.
But the usage of "man" and "men" to refer to a generic human or humans in general, irrespective of gender, continued well into the 20th century.
"Woman", meaning "an adult female human being", appears in Early Old English as "wifmon", as seen in the OED's quotation here:
eOE tr. Orosius Hist. (BL Add.) (1980) iii. vi. 60 Minutia hatte an wifmon þe on heora wisan sceolde nunne beon [L. Minucia uirgo Vestalis].
Thus, it is true that the "man" in "woman" is non-gendered, at least at the word was created.
As an aside, some other answers consider a word with both a gendered and non-gendered meaning, but primarily a gendered meaning, to be a gender-neutral word, which I find odd.