When the sun goes down is the common use and form for a setting sun (as oppose to the sun comes up for a rising one).
The image in my head is a person standing on the beach in the evening waiting to take a photograph of the setting sun.
Let's wait for the sun to come down seems to imply that you're waiting for the sun to catch up to you for some reason (as if you're ahead of the sun).
The image in my head is a person standing on the beach in the late afternoon and telling his/her friend(s) that they should stay until the sun reaches them and sets in the distance.
There seems to be no formal difference, just a directional thing. I guess if your back is to the horizon and you're not watching the sun there, it's coming to you. And if you're facing the horizon and the sun is there, then the sun is going from you.
On a side note, there are lyrics of songs with both versions.