Let's say I want to refer to a toy that I had when I was younger. Would it be incorrect to say "young me's toy"?
If you are using the noun phrase young me to describe yourself in the past, then adding an apostrophe and s to form the possessive would be normal.
Young me was quite precocious.
Young me's toys were all red.
Having said that, it's unusual to use young me in the first place. More typically, a variation would be used:
When I was a child, all of my toys were red.
But I can see young me (and the possessive) being used in the context of fiction, for example, if you go back in time and meet yourself. In that situation, you'd have to differentiate between your current self and your young self in some way. It's not unreasonable to think of that wording being the chosen device.
It seems grammatical, although it sounds weird unless it's meant humorously. But I have seen writers use the third-person when referring to themselves with such a construction, like young Manar's toy.
It depends on your standards of correctness. Strictly speaking, me is the accusative of a pronoun which is I in the nominative and mine (if anything) in the genitive. So, one might think, it should be He greeted the young me but The young I greeted him and the young mine toy (the latter being clearly impossible). If you think the substativation of the pronoun makes it indeclinable---and that there is some reason to choose the accusative form---, then I see nothing wrong with the young me's (nothing more than with The young me greeted him). But if you want to avoid making the pronoun indeclinable you might consider my young self's.