There really is no general rule. Language evolves, and the evolution is primarily influenced by the people using the word, and different communities have different ways of thinking, so the “unisex” solutions turn out to be different for different words.
There is a critical distinction to be drawn here between at least three kinds of gender-neutral language.
One is grammatical neutrality: this is easy in English but hard in many languages. For example, moon has no grammatical gender in English, but is feminine in French (la lune) and masculine in German (der Mond). Nevertheless, a few words in English sometimes take particular gender pronouns: earth, moon, and nature, for example, certain moral qualities (such as wisdom and justice), and certain forms of transportation (such as ships and automobiles), are sometimes feminine and take the pronoun she. You might describe this as personification. Another term for it, according to the Wikipedia article “Gender in English”, is covert gender.
A second kind is etymological neutrality: language that contains no possible ambiguity, because it avoids root words that could be mistaken to mean only people who identify as a particular gender. In English, we rely more and more on such terms: attendant, parenthood, letter carrier, not stewardess, motherhood, fatherhood, postman.
A third kind is connotative neutrality: language having grammatical or etymological roots in a gender, but nevertheless used and understood to connote nothing about gender: manslaughter, freshman class, maiden voyage, master key, fraternal twins, lumberjack, matriculate.
In the fishing industry, the gender-neutral term actually used most is fisherman, plural fishermen. The term is neutral in the first and third sense, but not the second. It is grammatically neutral, neutral in connotation, but not neutral in etymology, much like freshman and lumberjack.
There was a campaign in Canada to adopt the word fisher, but the women in the profession largely refused to have anything to do with it:
[F]ederal efforts to replace fisherman with fisher in government documents, coupled with a high-profile Supreme Court decision on native fishing rights, caused a riptide of dissent over what to call people who fish. To complicate matters, many women in the industry didn’t want their job title changed and insisted on being called fishermen.¹
It is certainly possible that the preference for fisherman as the gender-neutral term will diminish as time passes. Over the past thirty years, the use of many such words has become less common. As of this writing, fisher has not yet caught on in occupational or popular usage. Some academics and governmental agencies now use the term fisher, and this may eventually influence the public. See for example the conclusion of the recent Australian Broadcasting Corporation article, “Is ‘fishermen’ a sexist and exclusionary term?” But only time will tell whether fisher really will catch on.