Number in sentences such as
- Add the three numbers together to find the total.
is obviously a count usage.
But in sentences such as
- We have visited Paris a number of times. compare
- We have visited Paris numerous/several/many times.
'a number of' is a compound quantifier taking a plural-form count usage such as 'times' (a number of / a dozen / 17 times). It is a fixed expression, certainly not including a count usage concerning 'number' (*17 numbers of times). But best seen as a largely invariant [fixed] expression (it can be modified: a large number of times; a sufficient number of voters).
We can also have sentences with 'numbers of' (often 'large numbers of', etc) again followed by a plural form count usage such as 'times':
- Large numbers of tourists visit the island each year.
Again, 'numbers of' is an (in this case plural-form) fixed expression, another quantifier, certainly not including a count usage concerning 'number/s' (*17 numbers of tourists).
[English Grammar Today; Cambridge Dictionary has (adjusted):
We use the phrases a number of and numbers of with a plural verb
[and plural-form noun phrase] when we mean ‘many’ or ‘several’:
There are a number of things we need to discuss.
A significant number of people are ill with flu.
Large numbers of bees have died because of the cold summer.
'Large numbers of' and 'a large number of' are closely synonymous, like 'lots of' and 'a lot of' (again see English Grammar Today; Cambridge Dictionary). But the first two compound quantifiers here can only be used with count usages (a large number of/large numbers of visitors; a lot of/lots of visitors/rice.