Is −9 a smaller number than −8?
And is −9 a lower number than −8?
What is the difference between lower and smaller here?
My gut instinct as a PhD mathematician is that −9 is lower than −8, but not smaller than −8.
There is not a technically correct term without more context being given. If you are talking about the integer numbers (including positive and negative numbers) then I wouuld prefer to say that -9 is lower than -8. If you are talking about the magnitude of the numbers, rather than their place on the integer number line, then -8 is smaller and lower than -9 (although I would prefer to say smaller). As there is not a technically correct answer, these phrases are often used interchangeably in practice. Mathematical notation, such as -9<-8 and |-8|<|-9|, would need to be used to avoid misinterpretation.
"Less than" is also a good alternative phrase. It is the words intended by the < sign.
In summary, I would say that a small number is close to 0 and, in a context where negative numbers make sense, a low number is close to minus infinity. It is best to give more context of what kind of numbers you are talking about.
You can't argue with the whole English-speaking world, and the fact of the matter is they overwhelmingly prefer smaller to lesser or lower.
I think if pressed most people would agree that -1 is a 'bigger' number than -2. But negative numbers don't exactly have real-world correlates, so we all tend to be a bit vague on that one.
Even more vague - for those who know what imaginary numbers are - is the question of whether i is 'smaller' than 2i
LATER: I have no opinion on whether either of OP's examples is more 'correct' than the other. They're both fine to me. I'm simply making the point that of the two specific usages being asked about, on average people prefer to use the former. Here's a somewhat more specific NGram for those who still want to dispute that point...
My math teacher used to say that in this inequality 5 is smaller than 3, but 3 is less than 5.
Joking aside, only less than and greater than are uniformly understood as
One could argue just about anything with Google NGram:
I would generally say that ‘bigger’ refers to magnitude (distance from zero) whereas ‘higher’ refers to value (distance from negative infinity), but in context anything is possible. We don't tend to use ‘big’ or ‘small’ in mathematical contexts very much. If you want to be correct, I would recommend ‘greater’ and ‘less’. In fact, the signs < and > are read ‘is less than’ and ‘is greater than’ respectively.
In http://oi56.tinypic.com/2lieama.jpg 5 is smaller and lower than 3, but still greater.