You are right, the expression has its secrets and can 't be understood logically. Etymonline only says: "let alone" meaning not to mention is from 1812. No hint that it is difficult to see how these two words acquired their special meaning. I have never thought about this problem before, but this post makes clear that we have an etymological problem.
I doubt that we may find an explanation in etymological dictionaries as such fixed sayings easily slip through the net. But it might be that on the Internet someone somewhere has raised this question before.
Longman's DCE (let, no.14): let alone, used after a negative statement to say that the next thing you mention is even more unlikely.
- The baby can't even sit up yet, let alone walk!
From this example you can see that the formulations "not to mention" and "even less" both fit and have the same meaning.
If we wanted to try to express the idea of "not to mention" with "let" one might say: The baby can't even sit up yet. Let us not talk of walking.
The saying "Let me alone! has the meaning "Don't bother me!, Keep away from me!" So one might guess the meaning of "let alone walk" might be said in the sense of: And I don't want to touch the subject of walking. That's a thing that is still far away.
I think that some native speaker can explain our etymological problem better than I, it is my first attempt.
Added: I should add that in my view "let" is no imperative, but a past participle.
2. Not to mention, as in
We have no room for another house guest, let alone an entire family. [c. 1800]
I've just found this example of The Free Dictionary, taken from AHD, American Heritage Dictionary. They say: from around 1800. So it might be
that "let alone" meaning not to mention came up in America,