Cambridge has
Cambridge
spot:
a particular place
In the case where we say "let's go to that spot", we clearly have a choice. It is a matter of volition, so we would not choose an unpleasant place; this implies that in this usage the spot is at least acceptable, probably pleasant, and thus does not have a negative connotation.