So I've been trying to figure this out recently. There is a word like that in Russian, which is my native language, but I can't seem to find an English counterpart to it. In Russian, the word is "попаданец"(pronounced as something like pop-ah-dun-ets if you're curious) and it's basically a person(usually from present day, though the reverse happens also) who gets somehow transported to another time/universe/world by some means, almost always against their own will.
As an example, Fry from Futurama fits the bill, since he went from the year 2000 to the year 3000 against his will. However, if Doctor Who(or any other time traveler) travels in time he doesn't qualify. Although if his TARDIS malfunctioned and he ended up stranded in a parallel universe, you could argue that he qualifies for that title.
So the question is: is there a colloquial term to call those characters in English?