Are the phrases skill set and skill sets both correct?
As I see it, set implies a single set of related skills whereas sets can be taken to mean multiple sets of skills around different concentrations.
Skill is a noun, not an adjective. However in that phrase it is used as a noun adjunct, so it serves as an adjective.
Aside from that, I agree entirely. "Well, that matches my skill set" and "The two jobs need completely different skill sets" being valid singular and plural uses, respectively.
In reference to all of one's skills, the term 'skill set' includes all of a person's skills with none left out of the fold or on the sidelines. But if we wanted to convey a breakdown of skills into different areas and components, the term 'multiple' can be used to more clearly convey a plurality of sets, as in 'multiple skill sets'. In this instance, 'sets' is plural.