Subject-Auxiliary Inversion (with Do-Support) only happens with some adverbs.
Most adverbs don't indulge.
In particular, neither
are normal colloquial American English.
Adverbs can appear almost anywhere in a sentence, especially its beginning and end
(though different types of adverbs have different affordances),
and they rarely affect the word order of the rest of the sentence, no matter where they wind up.
One place where this does happen, though, is with negative adverbs;
but only with certain kinds of negative adverbs. So,
- Rarely do we see him around here any more
- At no time did he break a window.
are OK (though they seem rather old-fashioned to some),
- *For no reason did he break a window
- *With no tools did he open a window
are ungrammatical with inversion, though they're fine without it.