There is no perfect term for both the movement and the act of leaving others behind.
For a start, try relocate and relocation.
We relocated to another table.
Our relocation left three people behind.
Merriam-Webster's entry for the verb "relocate":
transitive verb : to locate again : establish or lay out in a new
place
intransitive verb : to move to a new location
Most dictionaries I consulted report "relocation" as the noun form with a corresponding meaning.
One caution: many words and phrases for this kind of movement have negative connotations due to the history of forced population movements or of white flight, the phenomenon of white families moving away from communities to avoid living next to black or immigrant families. Because the noun form does not signal whether people are relocating themselves or being relocated, rhetorical care is warranted. Consider the collocation "forced relocation," which refers to moving "large numbers of people under threat, planned and organized by governmental authorities, armed forces and/or militias." So you'll want to be careful with how you use and modify relocation.