I would have to say that this is all about context.
I failed to find references in major dictionaries, so I would have to quote other less recognized sources:
From The "subject" of the gerund | Grammaring - A guide to English grammar:
The doer of the action expressed by the gerund can be[...]unexpressed and understood only from the context
From Gerunds as Subjects | Grammar Quizzes:
Because a gerund clause is reduced (i.e., not marked for tense, person, or number, and the subject omitted) information must be understood from context.
(I am well aware that the second quote is about gerunds acting as subjects, but I think the two situations are similar enough.)
1) This reminds me of climbing Ben Nevis years ago.
"Climbing" is an action which only a human (and some animals) can do, and I would assume "this" is something inanimate, so the subject is clearly "I" instead of "this".
2) I told you about losing my credit card, didn't I?
Here, the subject is "I" because in the gerund there is "my credit card". If I phrased it this way, the subject would be "you":
2*) I told you about breaking your knees, didn't I?