'North Anglian' is inappropriate, because while Norfolk, England is in the North East Anglia region, this is not true of Norfolk, VA.
As 'Norfolkers' could be construed as insulting given the sound of the last couple of syllables, I'd say Norfolkites is appropriate: a search online throws up no results to suggest otherwise.
Are there many demonyms for relatively minor US towns? Obviously major cities like New York, Chicago, L.A. and San Francisco have them, but surely that isn't the case for every town, because there are just so many. Would people from Norfolk not just refer to themselves as Virginians?
(They might even smile when they say it: 10 kudos points for anyone who explains that reference without resorting to Google! :) )