No. It is nothing to do with the difference between leaving and arriving.
"Go" and "come" are used consistently in English, though the difference is sometimes difficult to explain.
"Come" is used when the speaker associates themselves with the destination: usually because that is where they are when they speak ("Please come to my office now."); sometimes because that is where they will be when the person arrives ("Will you come to the party tonight? We're going to be in the Starlight Ballroom."); sometimes because it is a place associated with the speaker even though they are not themselves going to be there at the relevant time ("I'm away this week, but you can come to our offices any time").
So "come to class" has the implication that the speaker will be at the class, or that it is a class that the speaker has an involvement in. "Go to class" does not have that implication. That is all.