Yes, Jack is refering to the game of golf — no metaphors involved.
In the game of golf, several people hit their balls toward the hole several hundred yards away. The balls each come to rest somewhere hopefully close to the hole. Golfers are very particular about people interfering with their ball. They don't want anyone to move it farther from the hole than it was and other golfers don't want anyone to move it closer to the hole than it already was.
So when somebody accidentally picks up a ball that is not theirs (it is sometimes hard to tell one from another) everyone gets upset and comes over to yell at them — he's just met a bunch of new people, but probably not in the way he would have liked.