Normally, when a noun ends in s and it is singular, you add 's to form a Saxon genitive.This is also normally true for Christian names (for example, St. James's park in London).
When it comes to surnames, the matter is less clearly settled. When reading about the novels that Dickens wrote, you can find both Dickens's and Dickens'. Note however that the apostrophe should always be added.
In the case you mention, I suppose that the reason for the lack of an apostrophe is that the name "Jeans" is used as an attribute of the term instability, meaning that it defines of which instability we are talking about, not something which belongs or characterizes Doctor Jeans. In other words, Jeans is used as an adjective and for this reason it does not change.