When someone is away from you and wants to be where you are, do they tell you I want to come there or I want to go there?
I have always thought that go has a slight connotational leaning to describing the journey, or leaving the origin. Come has a connotation to arriving, or the destination.
Thus, "I want to go to Paris" states that I want to make a journey, with Paris as the destination, but that the journey is important.
"I want to come to Rome" implies that the arrival in Rome is for a specific reason that is personally, emotionally important to the speaker.
Moreover, coming to Rome, as opposed to going to Paris, seems to imply that the speaker is addressing someone who is in Rome, and will be coming to them or going to Paris.
I believe that both choices are grammatically valid, but that each has a slightly different meaning invested in it.
The most common/natural expression here is probably along the lines of:
I wish I were there.
(Okay, so many people don't bother to use the subjunctive - as you should - and say was instead of were.)
A possible alternative is I want to be there - it's slightly less natural sounding to me though, at least in British English.
I want to go there. might be an appropriate response if someone mentioned a foreign country/holiday destination that sounded appealing to you. I can't think of any real-word situation in which I want to come there. would sound best.
I would have to say that there is a fairly simple and straightforward rule to this confusion. If you are presently at the location, then use "come" (as in: While sitting at a cafe in Rome drinking espresso "I would love to come back next year." or alternatively, "Excuse me, can you come over here"). But, if you are not presently at the location, to use "go" (as in: "Will you go over there." or "Hawai'i? I would love to go there.").