As stated in the title, I'm trying to get correct preposition — of/from/in. The sentence corresponds to the title of my work that addresses degeneracies that occur in FKK transmitter. However, this "in" somehow does not sound correct, and I just wanted to consult someone, since this is a title, and should be correct. I suppose that "from" would work, right?
Your title has problems other than preposition selection; but with or without those problems corrected, the preposition to use in the title is in. Neither of the other two is suitable.
Problems with the title “Reducing degeneracies in FKK transmitter” stem from lack of an article, or from use of singular rather than plural, or from incorrect reference to a transmitter instead of to its transfer function or response curve, or from degeneracies (specific instances of degeneracy) vs degeneracy in general. Thus, one might write
Reducing degeneracy in FKK transmitters
Reducing degeneracy in an FKK transmitter
but perhaps more properly (if I understand correctly) should say
Reducing degeneracy in FKK transmitter transfer functions