I'm having trouble understanding the ordering of the phrase "let alone"
For example: "I don't have a dollar let alone a dime" and "I don't have a dime let alone a dollar"
Or (from a similar question on this site): "I wouldn't go near a sting ray let alone pet one" or "I wouldn't pet a sting ray let alone go near one"
To me the last ones are correct, but other people seem to disagree.
In the first one, if you don't have a dollar it's seems you likely you couldn't afford a dime. So the last version would mean I don't have a dime so there's no point even considering a dollar.
And in the second example, it's similar if you wouldn't go near a sting ray then you wouldn't be able to pet it so there's no point in the let alone.
It seems like the "worse" thing should go before the let alone. I have a similar problem with "X never mind Y"
Could someone explain the ordering of these phrases?