In the industry, the accepted term is fisherman, plural fishermen. There was a campaign in Canada to adopt the word fisher, but the women in the profession largely refused to have anything to do with it:
[F]ederal efforts to replace fisherman with fisher in government documents, coupled with a high-profile Supreme Court decision on native fishing rights, caused a riptide of dissent over what to call people who fish. To complicate matters, many women in the industry didn’t want their job title changed and insisted on being called fishermen.¹
There really is no general rule. Language evolves, and the evolution is primarily influenced by the people using the word, and different communities have different ways of thinking, so the solutions turn out to be different for different words.