Grounds zero is unheard of in Google Books, in contrast to ground zeros and zeroes (both zeros and zeroes being acceptable pluralizations of zero).
(While the above includes many coincidental colocations, as at sentence breaks, almost all results after World War II relate to ground zero in miltiary or civil defense contexts, referring to the point on the ground below the detonation of a nuclear device.
A search on it in other corpora turns up, almost exclusively, titles from some of the laziest editors anywhere for articles or screenplays related in some way to coffee and used coffee grounds.
Ground zero, as with point zero, is a set phrase referring to a specific location. Zero is not a measure or count, nor a postnominal adjective; ground zero is not analogous to saying phases three and four of the project, or the succeeding poets laureate.
Consider, for example, from the August 1, 1946 issue of the Bulletin of the Atomic Scientists:
At Hiroshima, for example, persons in a concrete building 3,600 feet from ground zero…
They were 3600 feet from ground zero, not at ground 3600, or ground 0.68 miles, or ground 1097.3 meters for what it's worth.