If the boy was here he would wet the coils, he thought. Yes. If the boy were here. If the boy were here.
A paragraph from Hemingway’s The Old Man and the Sea (1952).
Grammatically, the difference in using was and were in these sentences is only that was is less formal than were. Both past tenses express the same present irrealis construction. See Huddleston & Pullum's Cambridge Grammar of the English Language (CGEL). One of them is not a "stronger" or more real, or less real, version than the other.
The less formal was is used more today, especially in British English, than 65 years ago, when Hemingway released The Old Man and the Sea. Yet, there are some places where most speakers still eschew was; if I were you sounds much better than the alternative, which is still considered nonstandard (CGEL). And such a usage as "If you was my boy, I would...” sounds on the border of hideous to many, who would call it substandard. Beyond that, saying why Hemingway uses was once and then were twice in short succession is literary interpretation, which is off topic on this site. It could be an issue of style, of impact, of nuance... It doesn’t have to be anything much.
To show this, one asks why does Hemingway write, in the same part of the story, both
"I wish the boy was here," he said aloud and settled himself against the rounded planks of the bow...
"I wish the boy were here and that I had some salt," he said aloud. Shifting the weight of the line to his left shoulder...
Strictly speaking, was is less formal than were, but readers will be hard pressed to find any appreciable difference beyond that.
In another answer, rjpond cites CGEL:
Preterite was, however, is widely used instead of irrealis were in these constructions, especially in informal style: He talks to me as if I was a child; I wish I was going with you.
(Cited by rjpond as Huddleston & Pullum's The Cambridge Grammar of the English Language (ch. 3, p86)).
When we read The Old Man and the Sea, we find out that the Old Man of the story, a rural fisherman, usually talks and thinks more "formally". For example, he "says aloud"
I wish it were a dream and I that had never hooked him.
Many/most speakers today would naturally use was, at least if it were/was not coupled with another clause.
Again, our fisherman says
"If you were my boy I'd take you out and gamble,"
and not the execrable but not unheard of "If you was my boy..."
So, our rural undereducated fisherman often speaks pretty dang good formal English. A possible
explanation of this is that the Old Man's utterances are rendered as close to his actual language (Spanish) as possible, and Spanish has a full blown subjunctive that English doesn't. The language of the Spanish speakers in Hemingway's For Whom the Bell Tolls is rendered this way. But whether this can explain the situation in the paragraph at hand (is there a Spanish equivalent of if the boy was versus if the boy were?)
is unknown to me, and even there is, it would be lost on most readers.
Then why use were twice immediately after using was? Proffered opinions include stressing that the boy is not actually with the man, as if the old man was "less aware" that the boy wasn't with him when he first cogitates. This seems unlikely.
Others offer that the change is for emotional impact. But we don't need a change in wording to do that. Saying the same phrase (with the same verb form) three times does that. Anytime Hemingway repeats himself, it's for impact. So he could have chosen were or was and stuck with it three times. But look at one alternative:
If the boy was here he would wet the coils, he thought. Yes. If the boy was here. If the boy was here.
Hemingway is too good of a writer to write if he was three times, or even just two times, in a row and have two instances of it end a paragraph. It's a matter of style and of good literary writing overall, to change to the more formal were.