Is there a term that can be used to refer to both the prefix and suffix of a word?
For example, unenjoyable. I'm looking for a collective word that described BOTH un and able.
I think the word you are looking for is affix, though that also includes "-fixes" other than just "suffix" and "prefix".
(linguistics) A bound morpheme added to a word’s stem; formerly applied only to suffixes (also called postfixes), the term as now used comprises prefixes, suffixes, infixes, circumfixes, and suprafixes.
Like the other answers mentioned, the general word referring to prefixes, suffixes and similar, is affix. However, Wikipedia also mentions adfix as joint name for prefixes and suffixes but not any other kind of affix:
It [infix] contrasts with adfix, a rare term for an affix attached to the end of a stem, such as a prefix or suffix.
See the definition of adfix in Wiktionary.