In the question What part of speech are "plus", "times", and "minus", we discover that plus is a preposition, and are left to assume that so is times, in phrases such as "five times six".
That seems to make sense to me.
However, I checked Oxford Dictionaries, and it states that times is a plural noun.
Not to be won over by the lesser sibling, I checked the OED.com and found, to my surprise, the same answer:
A. n. ...
II. A point of time; a moment in time; a space of time considered without reference to its duration; an occasion, an instance. ...
19. In pl. Preceded by a number (in words or figures). ...
b. Followed by a number or an expression of quantity: expressing the multiplication of this by the preceding number.
Conventionally represented by the multiplication sign: 4 × 5 is read as ‘four times five’.
How is times, in e.g. "Four times five is twenty.", considered a noun?