Letters typically correspond to phonemes - an "A" makes an "ah" sound. What sound does a hyphen make?
Accents or other diacritics aren't letters because despite the fact that they modify the pronunciation, they do not correspond to any phoneme themselves. Even if the argument can be made that a hyphen modifies how a word is pronounced (and I would argue that in this case it doesn't, as "uh oh" without the hyphen is pronounced exactly the same as "uh-oh" with a hyphen), that doesn't make it a letter.