Let's say I want to refer to a toy that I had when I was younger. Would it be incorrect to say "young me's toy"?
If you are using the noun phrase young me to describe yourself in the past, then adding an apostrophe and s to form the possessive would be normal.
Young me was quite precocious.
Young me's toys were all red.
Having said that, it's unusual to use young me in the first place. More typically, a variation would be used:
When I was a child, all of my toys were red.
But I can see young me (and the possessive) being used in the context of fiction, for example, if you go back in time and meet yourself. In that situation, you'd have to differentiate between your current self and your young self in some way. It's not unreasonable to think of that wording being the chosen device.