In my math textbook, I saw an author use “square and invertible matrix.” Which one is more correct: “invertible square matrix” or “invertibly square matrix”?
With my limited knowledge on Linear Algebra, I view it as such:
invertible square matrix
This means your matrix is both square and invertible. It has an order of n*m where n equals m, and it is also invertible.
invertibly square matrix
This implies that your matrix is square, and how square is it? It is invertibly square. I do not know if that even makes sense to say that something is invertibly square. Something can be almost square or perfectly square or permanently square, but I don't see how invertibly square could possibly mean anything.
Thus, I would say that using an adjective here probably means what you intended to mean.