Man in Old English could be either gendered or non-gendered. We inherited that ambiguity.
In Old English, man referred to both an adult male and a human being of either sex. Here is Stephen A. Barney in Word-Hoard: An Introduction to Old English Vocabulary, entry 8:
Mann serves for both "adult male" and "human being (of either sex)," in English; the other Germanic langs. adopted distinct words for the two senses: ModG Mann and Mensch "human being." The latter form occurs in OE (not in our texts) as mennisc (adj.) "human(s)," which survived to the 12th c. The OE terms which discriminate sexes are wer (Lat. vir) and wif (+ man = woman). ModG, like OE, has man in nom. (unstressed) meaning "one" (cf. French on).
Compounds include an early version of mankind, man-cynn. So what we have is a word that can refer to either adult males or human beings. Furthermore, it was sometimes hard to distinguish these uses. While the context makes telling them apart easily enough in this usage OED:
lOE Anglo-Saxon Chron. (Laud) anno 639 Þæs dohter wæs gehaten Ercongota halifemne, & wundorlic man. (This daughter was called Eorcengota holywoman and wonderful man)
Can you tell which man is meant in the next two uses? Testicles makes the second men obvious. The first manne (referring to men being hanged for thievery) is rather harder, unless you believe that only men could be hanged for thievery.
lOE Anglo-Saxon Chron. (Laud) (Peterborough contin.) anno 1124 Þet wæron on þa litle hwile ealles feower & feowerti manne, & six men spilde of here ægon & of here stanes. (There were in a little while four and forty men*, and six **men spoiled of their eyes and their testicles.)
Or how about the following usage? In the Old English Homilies, did Jesus became both man (an adult male) or man (a human being) for us? Either fits.
a1200 MS Trin. Cambr. in R. Morris Old Eng. Homilies (1873) 2nd Ser. 199 (MED) He bi-com man for us.
Also, to the extent we've been discussing a word in a language that still had grammatical gender, man in Old English is masculine. While grammatical gender should not be confused with biological gender (see Latin poeta, agricola, incola, nauta, all appear feminine in form but function masculine when modified by adjectives and most likely refer to men), grammatical gender doesn't help us distinguish semantically gender-neutral cases here.
So it would oversimplify to calling this usage gender-neutral OR strictly gendered, since the reading as adult male and the reading as human being may lead to ambiguity. Furthermore, such ambiguity doesn't exist for any word that refers to women - wif, bryd, faemne, cwene, and other words for women do not enjoy double-status as words referring to humans in general. Thus, while Old English man could be used to refer to human beings (including individual women), man (and the adult male it sometimes refers to) possesses a dual quality that woman and other female terms lack.