We have to be careful about how we use the term past tense. The verb go has no cognate past simple form derived from it. It has a kind of past participle gone. So if we want a past simple, meaning the past of what I go means, we have to resort to the past of another word, now rare in ordinary (non-poetical or archaic) language: 'wend', whose past form is 'went'.
But the use of 'been' in the way that puzzles you goes back a long way. There is the nursery rhyme, for example:-
Pussy cat, pussy cat, where have you been?
I've been to London to look at the Queen.
"I have been to London" is in the poem, not "I have been in London." It means almost the same as "I have been in London", but not quite the same, I should say. It has the sense of going to a place and coming back again in a short time.
So "I've been to Africa twice" means that I have visited Africa twice (perhaps on holiday, or business or for research). "I've been in Africa twice", on the other hand could mean the same thing, but it could equally means that I have lived in Africa for some significant period of time (longer than a long holiday, for example).