A metaphorical expression used quite often hereabouts is ghost of the past.
The Urban Dictionary definition is fine here:
ghost of the past
Horrific event [the memory of which one wishes] to be rid of, or person that did a horrible act towards you
I wanted to get rid of the ghost of the past, my little goat dying ...
The related lay the ghost of something (to rest) is relevant:
lay the ghost of something (to rest)
to finally stop being worried or upset about something that has
worried or upset you for a long time:
With one stunning performance, he has laid to rest the ghost of all his defeats last season.
[CDO]