I think this affects comma placement, right?
If it's an adjective phrase modifying harm, then I think it would be:
"potential, not actual harm"
If it's an adverb phrase modifying potential (by saying it another way, kind of like an appositive, but with an adjective instead of a noun), then I think it would be:
"potential, not actual, harm"
Saying potential another way is really the reason to put not actual in there in the first place, so treating not actual as an adverbial phrase seems to make more sense purpose-wise. Another way of thinking about not actual as an adverbial phrase might be:
"potential (and by that I mean not actual) harm"
However, I'm not really sure, especially since not actual certainly makes sense as an adjective modifying harm, which seems simpler. If a phrase could be seen as modifying an adjective or a noun, maybe for simplicity it should just be seen as modifying the noun?
I guess my possibly crazy-sounding hypothesis is that it's in the middle on some kind of spectrum between adjective and adverb and we should do some kind of primary purpose test to determine which one it is for comma placement? (I can picture English teachers everywhere cringing at that suggestion...) I think this is basically what I was thinking above when I suggested it might be an adverbial phrase. (The mental picture of cringing English teachers everywhere is why I think that suggestion might be wrong.)
Of course, this could be avoided altogether by just saying:
"potential harm, not actual harm"
But that's no fun. :)
So what do you think? Adjective phrase or adverbial phrase?