I think Participant might work, though that's more what you would call somebody who is in your meeting, rather than someone you will be meeting with.

I think Conferee might work.

Also, you could use participant, visitor, or caller, though participant implies they are already in the meeting, rather than someone you will be meeting with, and the other two imply the meeting will happen at your location.