I often notice this as a psychological response/defense mechanism that people do when afraid. They will talk tough about the thing they are afraid of. They will say that they will do horrible things to the thing they are afraid of, to compensate for the fear they experience. Is there a word or phrase or idiom for this?
You could use bluster, which fits the situation well:
angry or threatening talk or behavior from someone who wants to hide their fear or nervousness
How about bravado?
1. a. Boastful or threatening behaviour; ostentatious display of courage or boldness; bold or daring action intended to intimidate or to express defiance; often, an assumption of courage or hardihood to conceal felt timidity, or to carry one out of a doubtful or difficult position.
Source: Oxford English Dictionary (login required)
An excellent example of bravado is a mockingbird chasing a cat away from its nest.
Source: Words in a Sentence
The following phrases are quite close to what you seek:
put a brave face on/put on a brave face
: try to appear brave or cheerful or to be managing well in a difficult situation, when in fact you are frightened or unhappy
(The Free Dictionary)
put on a (brave) front
: To appear or make oneself seem more courageous, resolute, or dauntless than one really feels.
I could feel my knees shaking with terror before my commencement speech, but I put on a brave front and stepped out onto the stage to deliver it.
(The Free Dictionary)
This idiom fits well:
Whistling in the dark
It is used about someone who is making a noise in order to feel brave or safe.
As per Collins online:
If you say that someone is whistling in the dark, you mean that they are trying to remain brave and convince themselves that the situation is not as bad as it seems.
Bluff fits the bill quite nicely.
1a: a false threat or claim intended to deter or deceive someone : an act or instance of bluffing
Do note that neither defense mechanism, being afraid, nor talking are pre-requisites for executing a bluff. It can be used as an outright strategic offense.
Maybe not a perfect fit, but "saber rattling" comes to mind.
overtly and often exaggeratedly threatening actions or statements (such as verbal threats or ostentatious displays of military power) that are meant to intimidate an enemy by suggesting possible use of force
broadly : threatening statements or actions
This isn't necessarily used in fear or to bluff, but it can be. The idea behind this is to "make noise" to frighten the opposition from making any moves. If someone is talking tough*, but it's unlikely to result in any action, a response/goad could be "I hear a lot of saber rattling, but no action."
*I see that Caret tops already has that as an answer.
You ask if there is a word or phrase or idiom for someone who "will talk tough about the thing they are afraid of." Your own use of "talk[ing] tough" is apt, but I also think "talking trash" is a good idiom for this situation.
It's more commonly used in sports competitions, but the purpose is the same — to build oneself up by downplaying the scary nature (e.g., athletic prowess) of the thing that is feared.
Merriam-Webster defines it as
disparaging, taunting, or boastful comments especially between opponents trying to intimidate each other.
Intimidating the other and building up one's courage are two sides of the same coin.