I would still consider noon a noun (one that can by itself constitute an entire noun phrase/NP, so in this regard behaving more like a pronoun than like a regular noun such as day) in that context.
Related Linguistics SE post: Adverbs as NP pre-modifiers
In general, I believe many modern linguists consider it false to say that no adverb can modify a noun or noun phrase. (BillJ has left comments indicating that what's going on here is an adverb modifying a noun phrase.)
I found what looks like a relevant chapter "Adverbs, Noun Phrases, and Determiners" in the report A Lexicalized Tree Adjoining Grammar for English (The XTAG Research Group, Institute for Research in Cognitive Science, University of Pennsylvania, http://www.cis.upenn.edu/~xtag).
The authors state
Many adverbs interact with the noun phrase and determiner system in English. [...] Although there is some debate in the literature as to whether these should be classified as determiners or adverbs, we believe that these items that interact with the NP and determiner system are in fact adverbs.
They in fact list exactly, but describe it in a way that doesn't seem to account for the usage with noon that you asked about:
the class of adverbs that modify cardinal determiners. This class includes, among others, the adverbs about, at most, exactly, nearly, and only. These adverbs have the single restriction that they must adjoin to determiners that are card+.
Based on the description of determiner features in the chapter "Determiners and Noun Phrases", the class of "determiners that are card+" (or positive for the feature "cardinal") seems to consist only of cardinal numbers such as one, two, three.... All other determiners, such as all, both, this, the, a/an are shown as "card-" in the table on that page. This analysis does work for examples like "exactly three o'clock" and "exactly three twenty-five".
I think other examples to consider are "I ran exactly a mile", "At almost exactly the moment when...", "at exactly that second", "at exactly the moment", "at exactly this time". Most examples I can think of do use a determiner of some kind (although contrary to the chapter, they are not exclusively cardinal numbers) and you I guess could consider the adverb to be modifying the determiner rather than the noun phrase (although that feels a bit iffy to me with "exactly the").