Whom is your right choice here, Greg, meaning your gut was right. I do want to clarify why, though.
All the swans seen by other people whom I know have been white.
In this sentence, whom is a relative pronoun that is the direct object of the verb know. This pronoun is receiving the action of being known. I know him, and he knows me. It's hard to tell because there is an unwritten rule of English saying that all of the wh- words, like "what," "why," and "how," must go at the beginning, which is why it doesn't read, "I know whom."
The relative clause, whom I know, describes the noun people. That very noun is then the object of the preposition, by. In case you're confused about the purpose of whom in relative clauses, here are more examples.
The people whom I give apples to eat them up! (Object of prep)
The employees whom the employer pays hate the place. (Direct object of verb)
Just remember that there are predicate nominatives, which are responsible for recognizing that the noun or pronoun after the verb is the same as the subject. Verbs that follow PNs won't work in the passive voice.
Who are you! I am he. (Because the last sentence says that he and I are the same people, the pronoun after the verb is the same thing as I. He is now a predicate nominative. Also, "He is been by me," makes no sense at all. This tells us that "to be" doesn't follow objects ever.)
Who have you become lately? You became he. (By transforming into him, you and he are the same person, making he a PN. Also, in passive voice, "He is become by you," makes no sense. The verb know takes objects and will therefore make sense in the passive voice. "He is known by you," but never "He is become by you," because become never takes an object.)
Have a great day!