I'm looking for a word to describe using someone's own tactics against them. A formal word is preferable but a colloquialism will work as well.
The following phrases might be related to what you are asking about:
• fight fire with fire, “To respond to an attack with a similar or identical counter-attack”
• tit for tat, “equivalent retribution, an eye for an eye, returning exactly what you get” or “Having the characteristic of returning bad for bad”
hoist on one's own petard, or hoist by one's own petard, “To be hurt, or destroyed by one's own plot or device, of one's own doing which one intended for another; to be "blown up by one's own bomb"”
[Per comments I've removed one item and added the following one.]
• taste of one's own medicine, “(idiomatic, usually preceded by a) Harsh treatment inflicted on one who previously inflicted similar or equivalent treatment on others” [eg] “Some stars reportedly even want to investigate the private lives of tabloid editors, to give them a taste of their own medicine” [Time, 2001]
• two can play at that game, “something that you say when you intend to harm someone in the same way as they harmed you: When I found out that my husband had been having an affair, I thought "two can play at that game!"” [dictionary.cambridge.org]
• turnabout is fair play, “It is fair for one to suffer whatever one has caused others to suffer. So, you don't like being made fun of! Well, turnabout is fair play.” [idioms.thefreedictionary.com]
to beat sb at their own game is to use to your own advantage the methods by which someone else has tried to defeat you.
I.e., to use their own tactics against them.
Kipling, in 'Stalky & Co', wrote ..."Did you ever read a book about Japanese wrestlers? ... These wrestler-chaps have got some sort of trick that lets the other chap do all the work. Then they give a little wriggle, and he upsets himself. It’s called shibbuwichee or tokonoma, or somethin’." While Stalky's grasp of Japanese is not to be relied on, the term shibbuwichee is ooccasionally used in your context, e.g. here.