Should I say “I only have one distinct memory from the movie, that of a scene where …”, or would it work without the that of? If it wouldn’t work without the that of, what if I replace the comma with a colon? Would it be ok then?
It would work with or without the that of.
If the bit after the comma was an independent clause (it could be a sentence on its own), then a colon could be used.
You can also jump straight into the description with, e.g.:
You might want to replace "I only have one" with "My only..."
Otherwise, a colon is a good option: