The Blackwell Encyclopedia of Writing Systems defines a letter as “a graphic symbol that represents one or more phonemes of a language”.
In other words, letters correspond to sounds - an "A" makes an "ah" sound. What sound does a hyphen make by itself?
Accents or other diacritics aren't letters because despite the fact that they modify the pronunciation, they do not correspond to any phoneme themselves. Even if the argument can be made that a hyphen modifies how a word is pronounced (and I would argue that in this case it doesn't, as "uh oh" without the hyphen is pronounced exactly the same as "uh-oh" with a hyphen), that doesn't make it a letter. The vast majority of people would not associate "-" with any particular sound whatsoever in any circumstance, and the vast majority would not consider it a letter.
Some interesting discourse on what makes a letter may be found here, trying to disentangle the functional role of a letter from its representation from the very notion of it.