The observation behind this question is very nice. Use of this locution in indeed limited, to perfect tenses:
- Present perfect: I have been to London to visit the Queen
- Past perfect (pluperfect): I had never been to London before you took me
- Future perfect: I will have been to London by the time you return
You can clearly imagine putting the phrase into other tenses. For instance, by analogy of I have been silly to I am (being) silly, you could form *I am to London, *I am being to London. Similarly, you expect *I was to London (I was silly) and *I will be to London (I will be silly). But none of these are possible.
Note 1. It is wrong to think of to have been as having have as a main verb. If so, one would expect the possibility of the past perfect have had been (comparing to, say, have had it coming), and this is clearly impossible.
Note 2. There is a (rather outmoded English?) idiom to have been, without any destination/location specified. It is a euphemism for going to the toilet. Monty Python made a recurrent joke out of it in a skit based around an Agatha-Christie-style murder (the reluctance to say murdered led to numerous characters asking simply Has he been ... ?, to which the others would answer something like Yes, before lunch).