My gut instinct as a PhD mathematician is that −9 is lower than −8, but not smaller than −8.

There is not a technically correct term because if you are talking about the magnitude of the numbers, rather than their place on the real number line, then both smaller and lower would be correct. As there is not a technically correct answer, these phrases are often used interchangeably in practice. Mathematical notation would need to be used to avoid misinterpretation.

"Less than" is also a good alternative phrase. However, "less than" cannot be used to say that a number is small or low in an absolute sense. In this case, I would say that a small number is close to 0 and, in a context where negative numbers make sense, a low number is close to minus infinity.

My gut instinct as a PhD mathematician is that −9 is lower than −8, but not smaller than −8.

There is not a technically correct term without more context being given. If you are talking about the integer numbers (including positive and negative numbers) then I wouuld prefer to say that -9 is lower than -8. If you are talking about the magnitude of the numbers, rather than their place on the integer number line, then -8 is smaller and lower than -9 (although I would prefer to say smaller). As there is not a technically correct answer, these phrases are often used interchangeably in practice. Mathematical notation, such as -9<-8 and |-8|<|-9|, would need to be used to avoid misinterpretation.

"Less than" is also a good alternative phrase. It is the words intended by the < sign. 

In summary, I would say that a small number is close to 0 and, in a context where negative numbers make sense, a low number is close to minus infinity. It is best to give more context of what kind of numbers you are talking about.

My gut instinct as a PhD mathematician is that -9 is lower than -8, but not smaller than -8.

There is not a technically correct term because if you are talking about the magnitude of the numbers, rather than their place on the real number line, then both smaller and lower would be correct. As there is not a technically correct answer, these phrases are often used interchangably in practice. Mathematical notation would need to be used to avoid misinterpretation.

"Less than" is also a good alternative phrase. However, "less than" cannot be used to say that a number is small or low in an absolute sense. In this case, I would say that a small number is close to 0 and, in a context where negative numbers make sense, a low number is close to minus infinity.

My gut instinct as a PhD mathematician is that −9 is lower than −8, but not smaller than −8.

There is not a technically correct term because if you are talking about the magnitude of the numbers, rather than their place on the real number line, then both smaller and lower would be correct. As there is not a technically correct answer, these phrases are often used interchangeably in practice. Mathematical notation would need to be used to avoid misinterpretation.

"Less than" is also a good alternative phrase. However, "less than" cannot be used to say that a number is small or low in an absolute sense. In this case, I would say that a small number is close to 0 and, in a context where negative numbers make sense, a low number is close to minus infinity.