B, which uses "hadn't," is a better description of an event that has already ended. In this example, Simon feels exhausted because he was unable to get any sleep during the night. So the night is over, and we switch to a more perfect tense to describe what went on inside it.
A's use of "wasn't" probably falls under the Simple Past designation. "The baby wasn't sleeping all night" — you might come across this sentence in a book when the narrator talks about a year when insomnia was a bodily habit of the baby. But it's not specific about the time in which this took place. B talks about a recent and contained span, while A could refer to a longer one. For example, "last month the baby wasn't sleeping through the night." In the page linked below, glance at uses 3, 4 and 5.
In short, "had" indicates that an action has been completed. "Was" is a more general description of the past that can cover a whole time period. This example calls for a narrow description of when poor Simon had to stay up.
Hope that helps. Narrative tenses are tricky until you get a lot of practice.