"I don't think it's fair for me to be forever branded the other
man/woman."
More formally:
The person an adulterer has an affair with is the third party.
other man (n.)
A man with whom a married woman has an affair —usually used with the
E.G. Marshall plays a man who, believing himself cuckolded, plants a bomb in his own basement to destroy the Mrs. and the other man. —
Mike Flaherty m-w
The lover of a married woman or female partner. Lexico
other woman (n.)
A woman with whom a married man has an affair —usually used with the
m-w
The female lover of a married or similarly attached person.
Her husband left the matrimonial home to live with the other woman. Lexico
third party (n.)
A person or group besides the two primarily involved in a situation,
especially a dispute.
the involvement of a third party as an independent valuer
Lexico
"Adultery: What About the Other Woman (or Man)?" Looking at the
ethics of adultery from the other side... psychologytoday.com
Question
Can I sue the other woman for destroying my marriage?
Answer:
This is a very common question, but a pretty uncommon scenario.
Depending on where you live, you may be able to file suit against the
other woman (or man) through one of two types of civil tort claims – "criminal conversation" or "alienation of affection."
Lina Guillen; "Can I sue the other woman for destroying my marriage?", divorcenet
Whilst some people call a close friendship between a married man and a
female more than questionable, others define "being unfaithful" only
as having actual sexual intercourse with a third party. Petra Falk; Being the Other Woman: the Complete Hanbook for the
Woman in Love with a Married Man (2009)
In general, society looks at the other man or other woman as
being the responsible party in an affair. It’s understandable that
they become the target for the rage and anger the deceived spouse
feels. Cathy Meyer; "4 Tips For Dealing With The Other Man or Other Woman",
liveabout.com
It is generally recommended against naming the third party – it
can greatly raise tensions, as well as costs and timescales as there
are more parties in the proceedings. It has been far more usual
practice for Petitioners to refer to adultery being committed with an
unnamed man or woman. Wedlake Bell; "Naming the third party in an
adultery application for divorce"
How Do We Define Adultery?
... We might start by saying that what is called for are two
relations, that between a man and a woman who are married to each
other, and that between one of these two and a third party, who
might be married, or single. Edward Gallafent; Adultery and the
Female Star (2018)
For instance, Gn 38:24 defines a betrothed woman's sexual intercourse
with a third party as adultery deserving the death penalty (see also
Dt 22:23). A. Abasili; The Understanding of Adultery in the Hebrew
Bible (2016)
Respondents will often give acceptable evidence anyway, and this is
especially so if by giving the confession the respondent is able to
keep the name of the third party involved out of the suit, as is
now the norm.... Frances Burton; Family Law (2003)
It's also extremely unfair to the third party involved in the
affair who oftentimes gets hurt. E. Guevara; Unveiling and
Conquering Adultery (2020)
If the offending spouse is believed to have had an opportunity to
cheat, and if s/he seemed to be intimate with or fond of a third
party, the courts could consider this as adultery.... S. Sahni and
G. Jain; Internet Infidelity (2018)