I'm programming a math library and it never ocurred to me before now that most mathematicians say "translation, rotation, scale" to refer to these transformations. Problems arise when I want to distinguish the verbs "translate, rotate, scale" from the countable nouns "translation, rotation, scale". Is it wrong / strange to use "scalation" for the countable noun version?
In English, if you want to turn a verb into a noun, and there's not a noun that already exists for it, we commonly use the gerund formed by adding -ing. And sometimes, even if there is a such a pre-existing noun, we use the gerund instead.
Thus, we say
I like swimming and hunting,
and not (unless we are trying to sound ridiculous)
? I like natation and venery.
So scaling is the most natural noun to use in parallel with translation and rotation.
Scale is not a noun that means what you want. If you look at any dictionary, none of the definitions of "scale" refer to an instance of the transformation. The correct term is scaling. This is a Google search for scaling. It's sense 3 of the word, definition 2, and what you want is "scaling," gerund or present participle.
represent in proportional dimensions; reduce or increase in size according to a common scale.
This is a verb, but never a noun that refers to the action of scaling.