In theory that few doesn't violate any rule of grammar, but to use it would be to identify yourself as not a native speaker.
The first thing to notice is that the word that, in the sense of very, is only ever used in the negative.There weren't that many people at the concert/The horse wasn't that big. The only geographical area I know of where it is used positively is the north of England, specifically Yorkshire, where you might hear someone say, "I was that exhausted", "he was that handsome", etc. Everywhere else it's always found in a negative formulation.
However, "that few" is a combination I've never heard, either in the negative or the positive.
To make things even more confusing, the rule with very few is the reverse of that many. That is, it's only used positively. You can say, There were very few people at the concert, but not, There weren't very few people at the concert.