I always see whom used in questions. This is not a question, is a statement. Whom sounds correct to me, but I'm not sure if it is and why.
The "rules" I've seen around are all for questions, not for statements. In this case, "whom" seems the object to whom (hehe) the action of "being with" is performed. Is this correct?