Is there a special term for quiz bowl/knowledge bowl/trivia game-show competitors or people who otherwise spend their time remembering trivia for such competitions?
Quizzer may be used for a quiz bowl player, at least in some places. There are several examples in this Wikipedia article and related articles. This usage appears to be very informal, as I could not find a reputable dictionary that contains this definition. Quizzer does not appear to be globally embraced by quiz bowl participants. Most of the references I found were in India or on Wikipedia, with very few in North America and the UK. In any case, quizzer is not applied to TV game shows. Quizzer has a different usage in India -- see Yaitzme's comment.
Player is commonly used for quiz bowl competitors.
Contestant is commonly used for TV game show competitors.
Someone who is serious about trivia may informally be called a trivia junkie, whether or not they compete.