You are thinking of noun adjuncts, or attributive nouns.
Both Wikipedia and someone from Pearson Education state that traditionally, all attributive nouns are singular. However, the number of plural attributive nouns have increased over the years in a rather arbitrary manner. This means that there is no longer a set rule to determine whether the attributive is singular or plural.
Despite this, there is a rough guide that you can use.
The attributive noun tends to be plural in the following situations:
1: The singular form might lead to ambiguity
an arts degree (a degree in the humanities) as opposed to an art degree (a degree in fine art)
[I believe that this situation is what is referred to on wikipedia as "lexical restrictions", providing the example of "arm race" vs "arms race"]
2. There is no singular form of a noun (in pluralia tantum)
a customs officer
3. There is a need to denote variety
a soft drinks manufacturer [but] a car manufacturer
4. A topical issue comes forth, often in newspaper stories...
the tapes issue
the tapes compromise
the Watergate tapes affair
the White House tapes mystery
and other examples, including jobs cut.
[I am not entirely sure what the point of this fourth category is]
[Source: Pearson Education
, directly quoted apart from what is enclosed inside square brackets]
To this list I add my own:
5. [The attributive noun is plural if] it comes from the possessive form of the word
However, this is more of an explanation rather than a rule you can use. There are too many exceptions, and you are unlikely to know the origins of the word. It doesn't make sense that it is teachers union but not students council.
Note that we often "choose by ear and it doesn’t matter (employee lounge, employees lounge)" (Chicago Manual of Style).
I would argue that it would be easier to just look it up every time and memorise words instead of trying to find a rule or categorise them. Even as a native speaker, I was not entirely sure whether it was "sport medicine" or "sports medicine" before looking it up. If you find yourself completely lost and without internet, go with the singular form because it is the traditional form and more common.