You are right that is ambiguous. The difficulty comes because old here is being used as an opposite of new rather than as an opposite of young, but both meanings of the word "old" could be applied here in a way that would be valid.
We use context automatically to disambiguate in these situations, and there relatively few situations where you would care to be describing your friend who is the oldest age, so this doesn't tend to be confusing. "My oldest book" is much more ambiguous because it's easy to think of situations where you would be describing both the book you've had the longest and the book that was written/printed earliest.
If such a need did arise, you would say "the" instead of "my" (e.g. the oldest friend I have.) Logically, this could mean the exact same thing, but people would pick up the fact that you were using a different phrasing for a reason.
The web series Jake and Amir pokes fun at this ambiguity in the English language in Jake and Amir: Cheryl
Jake: Your best friend in the whole world? I've known Amir for 5
years, and I've just heard of you.
Cheryl: Yeah, we met yesterday.
Jake: You said you were his oldest friend.
Cheryl: I'm 50. Thank you for saying I look fantastic.