One relevant piece of data (although I wouldn't say it's conclusive) is the case of the singular second person pronoun thou/thee in archaic English, since this pronoun had distinct forms for the subjective and objective cases and was used in the vocative fairly often.
When it was used in the vocative, the nominative form thou seems to have been used, as demonstrated by this quote from Shakespeare:
Kent. Thou whoreson zed! thou unnecessary letter! (King Lear, 2.2.35)
Similarly, back when ye was still used as the subject case form of you, the sources I have found indicate it was also used for the vocative (The English Language; Its Grammar, History and Literature, by John Miller Dow Meiklejohn), although as ye passed out of use the situation seems to have become less consistent (An English Grammar, by Eduard Adolf Maetzner, mentions some grammarians who describe ye being used in the nominative and you in the vocative).
I think it's fairly unnatural to use a non-second-person pronoun in the vocative, so I don't have any strong intuitions on "dear he" vs. "dear him" and I'd guess that different people would have different preferences here. Guifa has left some comments suggesting "Dear me" sounds better than "Dear I"; I guess I'd agree, but I can't think of a situation where I'd use either and both of these options sound awkward to me (I think people normally use second-person pronouns to talk to themselves, as in "You've got to keep going!")
I'd recommend going with the nominative "Dear [whoever]."
Another piece of data may be that in present-day German, which has retained case to a greater degree than English, the adjective corresponding to "Dear" is put in the nominative singular form when addressing a letter: one writes "Lieber [name of a male]" at the start of a letter in German rather than using dative "Liebem" or accusative "Lieben".
Not all languages with cases deal with such situations the same way: in Ancient Greek, it seems that the receiver of the letter might be given in the dative case at the start of the letter (according to Alexandre Daubricourt's answer to the following Latin SE question: Are there any surviving Ancient Greek letters (epistolary)?).
English is more closely related to German than it is to Greek, but even closely related languages can sometimes use cases differently, so the German data is not conclusive with regard to English—it only suggests that the nominative case is a plausible candidate in this kind of context.