Object refers to affection, the object of his affections, that is his friends. The idea is that it didn't really matter if the friends he had suited him. The friendships he developed were left to chance, kinship, and time. In the end, they had grown on him like ivy does on a tree.
Here's a longer extract from Stevenson's book that gives a little more context:
- It is the mark of a modest man to accept his friendly circle ready-made from the hands of opportunity; and that was the lawyer’s way. His friends were those of his own blood or those whom he had known the longest; his affections, like ivy, were the growth of time, they implied no aptness in the object. Hence, no doubt, the bond that united him to Mr. Richard Enfield, his distant kinsman, the well-known man about town.