How do I say this correctly? I can't figure it out. Do I have it correct already? I am trying to say that there are a lot less people. The opposite would be "so many more people".
The correct way to say this is far fewer people.
I think that a sentence like "There are so many fewer people here than usual" is acceptable, conditionally.
The word so is emphatic. Therefore so much less or so many fewer should be avoided when the usual quantity is not great or numerous to begin with.
For instance "The paper cut in my pinky hurts so much less today" basically states that it doesn't hurt any more, but in a melodramatic way.
If there are only eight people here usually, then the emphasis in "so many fewer" is inappropriate because there can be at most eight fewer, and eight is hardly "so many".
A good rule of thumb may be: if the larger of the two quantities being compared cannot be emphatically described as so much or so many, then the smaller quantity cannot be so much less or so many fewer.
There were so many people here yesterday, and today there are so many fewer, in comparison.
I have to admit that when I saw the title of this question, I thought "I don't like that phrasing". And although I don't particularly like so many less people either, I thought it wasn't quite so odd.
But they're both relatively low occurence counts for what one might expect to be a concept that would often need to be put into words. After all, so many more people gets over 40,000 hits. I think the reason people tend to avoid these forms is simply because we don't like the "disjunct" created by placing many and either fewer or less next to each other. Graphically speaking...
...which simply backs up comments suggesting OP should stick to the alternative he came up with quite naturally in the question text - there are a lot less people.