It depends on the context. Consider:
Most of the money was raised from Mets fans and Yankees fans, who inhabited the same city and experienced the same concerns.
The priest and the penitent, who occupied the furthest confessional, could not possibly have heard the muffled gunshot. The would be useless as witnesses.
John, Paul, George and Ringo, who would later become famous as The Beatles, were at that point playing in an obscure German nightclub.
Without the surrounding words to give context, the who clause typically modifies the noun that it follows most closely. However, it can be made to "reach back" quite a way, especially if the list is perceived as a group.
On the other hand, consider again our friends the Smiths and the Thorpes, with some additional detail that would typically apply to only one party:
- We dined with the Smiths and the Thorpes, who had just moved into their new house, and who ordered like their bank account was scraping rock bottom.
I don't think that there's a general rule, since the perception of (say) John, Paul, George and Ringo as a group will also depend on the reader's prior knowledge. Same thing for knowing how people might act at an expensive restaurant, buy a house, share a bank account and so on.
You can sometimes can make use of words to group or separate the elements of the noun list, e.g. "same city", "famous as The Beatles", etc. But someone unfamiliar with the Catholic church might not grasp immediately how the priest and the penitent occupy the same space.
If in doubt, it's probably because your own prior knowledge is not grouping or ungrouping the nouns clearly. You can reasonably infer that your readers will also have trouble, and that you should rephrase what you are hoping to convey.