Unlike a lot of English, it turns out there is in fact a definitive source to start looking to for an answer to this. The term was invented and trademarked by the Open Source Initiative (OSI), and they claim to maintain the definition for exactly what it means.
So how do they do it on their own materials? Well, from poking around their website, I see two common methods:
- open source. "Open" is capitalized only if it starts a sentence.
- open-source. (Same comment).
The only time you see Open Source is when its part of the title of their organization name (iow: followed by "Initiative").
That being said, occasionally in the wild you will see open-source software abbreviated as "OSS". I also often see both words capitalized. Personally I tell myself that is because in those circumstances they are talking about it as a whole, thus personifying it, thus it gets its words capitalized just like a proper name would. That's probably just a rationalization though. Still, a run through Slashdot shows me that when used as an adjective term, it is pretty much always lower case.
So if I were to come up with rules based on the authority of the OSI and common use among practitioners it would be the following:
- open source or open-source when used as an adjective. (The hyphenless version is more common)
- Open Source when used alone as a noun (or occasionally a verb).
- OSS when abbreviated (to mean "open source software").