These are all considered "minced oaths". "Oh, man" could be replaced one-to-one with a Commandment-breaking "Oh, God" or "Oh, Jesus", which are generally shortenings of some prayer for strength.
An alternate etymology is that of friendly condescension; the terms can be literally used when talking to one's peer, junior, or brother: "Oh boy, you've screwed up this time" could be used when talking to your son or one of his friends after they break a window. The terms have evolved out of their literal meaning to simply become an exclamation indicating exasperation at a situation, even when you're not really referring to anyone else in particular.
The gender bias of the interjections is largely reflective of the historical gender bias in English-speaking cultures throughout the evolution of the language. There has been some insertion of the fairer sex in interjections: "oh girl" and simply "girl" are heard in informal conversation nearly universally now, virtually always when speaking directly to a "girl". Even then, there's no small amount of chauvinism in the term; the word as an identifier for females in general stems from the general male preference for youthful-looking women.