As the title says. I am wondering what is grammatically correct .
"non linearized" vs "nonlinearized" (or maybe "non-linearized")
Same for
"non linear" vs "nonlinear" (or maybe "non-linear")
Neither the one nor the other.
Usually, in the words beginning with "non", this one is connected with an hyphen to the main word.
However it is ... non-essential in a mathematical book, just better in a non-technical text.
According to a couple of dictionaries, "non" is not an English word but only a prefix. The OED does have it as a word, but not with the meaning apparently intended in the question.
When in doubt, consult sources/peers in the field in which you are working. The OED lists the main term in question as "non-linear," but if your (reliable) peers disregard the hyphen ("nonlinear"), you should follow their example. I'm sure that the Journal of Nonlinear Science would agree.
In the future, if you need a quick and dirty answer to solve an alternate spelling dispute, you can use the Google Ngram Viewer.