In [the] 1850s, a baseball club might consist of 40 members. By 1868, it was said that a club would have their regular professional ten, an amateur first-nine, and their "muffins".
Do ten and first-nine imply the number of club members?
If that's right, can I say that the club was smaller in 1868?
To answer your question, yes, "ten" and "nine" refer to the size of the team.
This sports terminology is not used very much in modern American English, but it is a common occurrence in British English. For example, the second-tier team for an English cricket club is called "the second XI" (where XI is Roman-numeral 11).
The numbers used refer only to the size of those teams: ten in the professional and nine in the amateur (the difference being that a professional team were permitted a spare or substitute player).
The club altogether consists of these teams along with the "muffins". Note in that link, further detail of how some clubs had a 'second nine' by the 1880s, under the glossary entry for 'nines'.
Finally, the phrasing of the original text implies that this is not one specific club but a general description of how clubs were sized in those respective years. The implication for the 1850s is that forty members would be near to an upper bound and most would not have nearly that number (commensurate with the history of most sports, where a club was often just one team) while by 1868 clubs were regularly putting out two teams in addition to a noteworthy number of other enthusiasts and lower-ability players.
"1868, it was said that a club would have their regular professional ten".