"So" in this form refers to the previous statement. You could rewrite this paragraph as follows:
What does a bookworm have in common with a black-tufted marmoset? They both like a little quiet, say scientists in the Royal Society journal Biology Letters. [Marina Duarte et al., "Noisy Human Neighbours Affect Where Urban Monkeys Live"]
The implication is that the previous sentence could be taken as an opinion, which in most formal writing has little weight unless backed up. By saying "Or so say" immediately following the statement, the writer or speaker is inferring that while the speaker or their audience may not agree with this statement, or have enough knowledge to form their own opinion, the smart, educated, experienced people identified after those three words do agree with it. As such, it is a persuasive device to get you to think that both bookworms and black-tufted marmosets like quiet, because scientists said so in the Royal Society journal, Biology Letters.