The main verb in your question is "know", so it is "The man whom I know to be unhappy", just as it is "The man whom I know".
It gets more complex if you replace 'to be' with 'is', as there are several possible meanings. "The man who, I know, is unhappy" is equivalent to "The man who is unhappy (I know it)", so whom would be wrong. "The man, whom I know, is unhappy" = "The man is unhappy: I know him (not he). Without any commas, or (just as wrong) with a single comma after 'know', ambiguity makes it impossible to say what the pronoun should be (unless the rest of the sentence makes it clear). Moral: punctuation is important, and don't lazily cut "to be" down to "is" unless you are clear about how you are changing the meaning.
NB Precise Edit's answer (quoted by Lauren), leaves out all the commas in this phrase, so isn't helpful.