On Wikipedia, the usage is "filter bank" exclusively.
A search on Google Scholar returns essentially the same number of both spellings.
This is for a scientific document about digital signal processing, so I want to know which is correct.
A Google Ngram showed that filter bank is used more widely:
Because of this, I recommend using filter bank.
Wiktionary gives "filterbank" as an alternative of "filter bank". Personally, the correct spelling is "filter bank" (with a space inbetween) because that's the original.
The only searches on Google that rendered "filterbank" was the name of a product, so I would advise you to use "filter bank" for your paper.