The term "court-martial" refers to military institutions whereby those accused of breaking the law are brought to trial.
This word is used in both the way we would use "trial" to refer to the proceeding ("I went to trial/I face a court-martial"), but it is also used like a verb ("I was court-martialled"). I have seen reference to "courts-martial" as is standard for pluralizing hyphenated terms.
Usage of the words "court" and "martial" is fairly simple. The former referring to the institution and/or the physical location, while the latter is a modifier to make something refer to war/military/fighting: martial arts, martial law, etc.
It is the combination of the two is where I would like some additional insight. It seems to me that "Martial Court" would be the natural way to link the two concepts, but here we are with a hyphenated word...and I would otherwise expect a more formal term than "court-martialling" to use as a verb for so severe an action.