The Blackwell Encyclopedia of Writing Systems defines a letter as “a graphic symbol that represents one or more phonemes of a language”.
In other words, letters correspond to sounds - an "A" makes an "ah" sound. What sound does a hyphen make by itself?
Accents or other diacritics aren't letters because despite the fact that they modify the pronunciation, they do not correspond to any phoneme themselves. Even if the argument can be made that a hyphen modifies how a word is pronounced (and I would argue that in this case it doesn't, as "uh oh" without the hyphen is pronounced exactly the same as "uh-oh" with a hyphen), that doesn't make it a letter. The vast majority of people would not associate "-" with any particular sound whatsoever in any circumstance, and the vast majority would not consider it a letter.
Some interesting discourse on what makes a letter may be found here (What is a letter?), trying to disentangle the functional role of a letter from its representation from the very notion of it.
What is a letter? Some interesting observations that letters aren't solely defined by their phonetic nature, as letters are still letters when used in mathematical notion, for example. But it seems clear there is a strong link between letters and phonemes, which a hyphen does not have.