Cambridge has
Cambridge
spot:
a particular place
In the case where we say "let's go to that spot", we clearly have a choice. It is a matter of volition, so we would not choose an unpleasant place; this implies that in this usage the spot is at least acceptable, probably pleasant, and thus does not have a negative connotation.
Some support for this view may be found in google ngram. Here are five sorts of spot. In contemporary usage, these negatives weigh less than these positives .