It kinda doesn't make sense to me because it's something that cannot disappear like moments or a person.
This is synecdoche, using a portion of something to refer to the whole. A less metaphorical sentence might be "I miss [the activities that led to those good] memories, man."
If you insist on a literal interpretation, you might miss memories that you no longer have, but remember having. For example, if you can no longer remember how your dead parents smells, but you are nostalgic for the time when you could recall that scent, you might "miss the memory".
If someone has amnesia, alzheimers, brain damage or such like, they could miss memories, but I suspect that in this particular case, the person misses making the memories.
Actually, 'miss memories' isn't the correct phrase. It can be 'Lost memories'. BUT, miss/lost memories refer to the result of a memory loss or brain damage. Hence, they cannot be used in sentences.