Of course, the word "center" has some uses that do not overlap with "middle" at all, like in the names of organizations ("The Center for X").
There are also a number of fixed expressions where the words obviously cannot be interchanged, like "the middle of nowhere", "the middle ear", "the Middle East".
Differences that I think exist between "center" and "middle" in other contexts:
The word "middle" is used more often than "center" when referring to time. You can say "the middle of the day" but we usually don't say "the center of the day". The phrase "in the middle of" is often used to refer to something that is unfinished and still in progress.
The word "center" feels more appropriate to me than "middle" in certain contexts where you are talking about something that is "centered" in multiple dimensions, like the center of a circle: "the exact center of the bullseye" sounds better to me than "the exact middle of the bullseye".
I think my advice here is supported by the results of a Corpus of Contemporary American English (COCA) comparison that I looked at between "the center of" and "the middle of". The top collocates that were used with "the center of" more often than "the middle of" included things like "galaxy", "universe", "attention", "labyrinth", "planet", "cosmos"; the top collocates that were used with "the middle of" more often than "the center of" included "night", "afternoon", "decade", "July", "January", "sentence", "century", "divorce", "interview", "grade", "term", "concert", and "wilderness".