I am trying to describe someone's reactions in a situation. At first I used the word overreact, but then I realized it is not only the person's overreaction but also other kinds of inappropriate or unexpected reactions. Then I came across "misreact" which seems to be the right word, except that I am not sure if it is a real word.
You are correct, you can use word misreact.
The prefix mis- conveys that something is wrong. From the Oxford Dictionary:
1. (added to verbs and their derivatives) wrongly. misapply
1.1 Badly. mismanage
1.2. Unsuitably. misname
A few references for misreact:
Ernest Hemingway's The Sun Also Rises: A Casebook, by Linda Wagner-Martin, ed.
If you are looking for a word with an actual dictionary definition (although I'm not certain that misreact is inappropriate), the closest in form and meaning I can think of is miscue:
: to make a miscue
2 : MISTAKE, SLIP
the slightest miscue could make the trapeze artist lose his grip and fall to the mat below