The problem with using formulary for a collection of mathematical formulae is that a formulary already means a bunch of prescriptions, like your health insurance company’s formulary.
Since this is a technical sense, it may be better to return to the original Latin form, which would be a formularium. Of the -arium suffix, the OED writes:
-arium, suffix of sbs. ad. L. -ārium ‘thing connected with or employed in, place for,’ orig. neuter of adjs. in -ārius. Of this the regular adapted form in Eng. is -ary: see -ary1, but the unchanged L. form is used with a few terms of classical and ecclesiastical antiquities, or of learned use, as caldarium, frigidarium, sacrarium, honorarium, herbarium, the last of which, being in general use, has probably popularized the use of vivarium and aquarium.
Other words of that ilk include the easily recognized. . .
- antiquarium for antiquities
- cinerarium for ashes (note that cinerary was already taken)
- formicarium for ants (formicary means the same thing)
- leprosarium for lepers
- muscarium for flies
- ossuarium for bones (ossuary means the same thing)
- ranarium for frogs
- rosarium for roses (note that rosary was already taken)
- sanitarium for health (note that sanitary was already taken)
- unguentarium for ointment
- vestiarium for garments (the rare vestiary means the same thing, but so too does vestry)
- viridarium for greenery
When J. R. R. Tolkien needed a word for the collection of legends he was building up, he chose legendarium. Yes, it is fancier than the normal -ary suffix that would yield legendary, but that word was already taken.
In the same way, since a formulary is already taken, you might choose a formularium instead.