Many words borrowed from another language do not retain their prior language's number or case structure in English. Singular or non-count bacteria and criteria are frequently used with verbs in a singular form, and are at least somewhat assimilated into standard forms of English. Singular or non-count millennia and phenomena rarely appear with verbs in a singular form, and thus appear to still be anomalies.
It is possible to refer to a single strain as a bacterium (the Oxford English Dictionary lists bacteria under bacterium). Note, however, that bacteria is far more common in use than bacterium (Ngram):
The Corpus of Contemporary American English also has several results for "bacteria _VVZ" (bacteria with a verb ending in -s). The official status of some of these examples leads me to believe bacteria is an acceptable form of the singular in many uses:
The bacteria grows under fluorescent light. (Cain Burdeau. "LA scientist's oysters safe from oil, but pricey." Associated Press, 20 Aug. 2010.)
Bacteria grows, you ingest it, you become ill. It's no secret. ("Behind the Counter - Restaurant Filth Exposed." ABC Primetime, 27 Sept. 1995.)
What's frightening is that the vibrio bacteria occurs naturally in warm seawater, unrelated to pollution. (ABC 20/20, 9 February 1990.)
But if we do know that bacteria causes, you know, gum disease and cavities, why not just do something right there? ("Potential links between chronic disease and bacterial infections." NPR Science, 26 February 1999.)
AP, ABC, NPR - a lot of news organizations use bacteria with a singular verb, especially in spoken contexts. That leads me to conclude it is widely accepted in standard language, even if some prescriptivists still wince upon hearing it.
Criteria has become more common through the 20th century.
It has also been used with -s verbs in publication and speech, with several significant results from COCA:
For example, the first criteria includes the use of creatine and artificial low-oxygen living environments. (Matthew Mitten. "Is drug testing of athletes necessary?" USA Today Magazine, vol. 134, iss. 2726, Nov 2005. )
The selection criteria includes academic achievement, leadership and community involvement, and students must graduate from one of the participating senior high schools in Brazoria, Chambers, Fort Bend, Galveston, Harris, Liberty, Montgomery or Waller counties. ("Katy American Little League^s Astros take first place in 8-year-old play." Houston Chronicle, 12 July 2000.)
The same criteria applies to next month's debates. (PBS NewsHour, 28 August 2019.)
So at least in some official uses, criteria used with a singular verb is acceptable.
The singular form remains more common, especially near the turn of the millennium.
No significant results for millennia with -s verbs in COCA. In my experience, millennium is still overwhelmingly used for the singular.
The Ngram indicates more recent growth in the use of phenomenon compared to phenomena.
In COCA, there are a couple of examples of publications using phenomena with an -s verb, but far fewer than for bacteria or criteria. I suggest that this remains uncommon, and phenomenon remains generally recognized as the singular form of the word:
I think it's fair to say that the phenomena exists from just off the coast of China all the way to a few hundred miles from the coast of California. (PBS News Hour, 13 November 2008.)
The tracking phenomena proposes that these V0 2 max levels at age 20 project the individual to be in the low-risk CVD category at age 45. (Bradley Cardinal. Cardiorespiratory fitness and physical activity behavior ..." Physical Educator, vol. 52, issue 4, winter 1995.)