Is there one? For example, "Bill was a hipster who only enjoyed [very strange and specialized genres of] music".
Perhaps recherché: unusual and not understood by most people (Merriam- Webster).
This word has a spectrum of meaning, from "exquisite/choice", through "affected", to "pretentious/overblown".
Would you be happy with such ambivalence? I think, from your quote, you might...?
I think the meaning is (a) well-known, and (b) has a wide application, so I will skip the definitions.
It is specifically applicable, given your description. The New American Dictionary states:
Obscure often expresses dissatisfaction at one's inability to identify something.
genre itself is specific (or specialised1), and that you are going to give a [specific] genre is the expectation of the listener.
You are trying to identify a genre, but you can't. So you want to express all that in one word.
Bill was a hipster who only enjoyed obscure music
Bill was a hipster who only enjoyed recherché music
The foreign word is foreign, unknown, obscure (sorry). You will have to stop and explain what it means to the other person.
- Your title 'specialized genre' is redundant because genre is specific. Your description is completely understood.
As well as Margana's excellent answer, you could try arcane, recondite or esoteric.
In addition to all the good answers: