I don't agree that using neither or either in this scenario is correct: you use those in the case of a negative statement, but the statement in question is positive. If the statement was the negative version:
We don't go on vacation often.
Then the response would be "We don't either" or "Neither do we" or (colloquially) "Us neither".
But when the statement is positive:
We rarely go on vacation.
Then the response should be
So do we or
We do, too or (colloquially)
Granted, few will bat an eye, and everyone will know what you mean, if you respond conversationally to "We rarely go on vacation" with "Neither do we". It is easy to flip the original statement into the negative grammatical form "We don't go on vacation much".
But still, technically the reply is nonsensical, because it is effectively "Neither do we (rarely go on vacation)". The original statement is affirmative, and it doesn't make grammatical sense to append a "neither" to an affirmation.