Most co-ed fraternities are called "fraternities" (though I do know of one co-ed sorority; it was originally women-only, and retained the label "sorority" when it opened its doors to men).
However, there are some serious problems with your example:
- The phrase "fraternity of scholars" is already well-attested, and almost never refers to a Greek organization; rather, it's generally either figurative, or referring to co-ed groups like Phi Beta Kappa.
- Note that Phi Beta Kappa, despite being a fraternity, and despite being named with Greek letters, is not a "Greek organization", and will not evoke the Animal House image that you seem to have in mind.
- Even without that phrase, when you talk about mathematicians forming a fraternity, people are likely to think of a professional fraternity. There are actually quite a few math fraternities, and most of them are co-ed, but (like Phi Beta Kappa) they are not "Greek organizations" and will not evoke a Animal House image. Even readers who are not familiar with any math fraternities, specifically, are likely to picture something along these lines: there exist chemistry fraternities, engineering fraternities, and so on, so a math fraternity is a natural idea.
- More generally — no co-ed fraternities are "Greek organizations". There's no gender-neutral way to evoke the Animal House image.
- A large proportion of Americans have belonged to fraternities and sororities that are Greek organizations — and the majority of these do not haze. You should consider whether you really want to turn off a good chunk of your readership by presupposing that they conform to a negative stereotype. (It's a bit like writing, "When women mathematicians get together, their catfights are […]" . . . no matter where you might go with it, you've lost a chunk of your audience. Though to be sure, a hazing joke will not be anywhere near as offensive as a catfight joke.)
If your goal is just to set up a joke about mathematics-themed hazing, I'd suggest writing something like, "When mathematicians haze each other, it's by […]". Your readers will get that it's a joke. It doesn't require lots of setup.
(By the way — "brotherhood" can also be gender-neutral, but it never conveys the specific idea of a Greek organization.)