It has nothing to do with informal and formal. It is simply right or wrong. The key distinction boils down to the difference between verb moods in complex sentences, that is, sentences that have a main clause and at least one subordinate clause. The use of were and weren't in the subordinate clauses depends on the reality or truthfulness of the subordinate clause. If it is true, then the indicative forms was and wasn't are in order. If it is not true, i.e. counterfactual, then the past subjunctive forms were and weren't are used.
Let's look at some examples. One of the easiest to deal with is subordinate clauses where the verb in the main clause is some form of wish. The semantics of these sentences imply that the subordinate clauses are not true, so the past subjunctive forms are called for:
I wish I were able to take that job. (But I am not able to take the job.)
Do you wish she weren't on call tonight? (But she is on call tonight.)
The other common pattern is with if-then sentences. (Note: then may be implied if it is not always verbalized, and the then clause may precede the if clause in that case.) If the main and subordinate clauses are both true or not known to be true or false, then the indicative verb forms are used:
If I wasn't asked to help, at least I was willing to help. (And I wasn't asked to help.)
If they are prepared, then they will come out on top. (And they may or may not be prepared--we'll see.)
The most common pattern that people get tripped up on are if-then sentences where the if clause is counterfactual. In these cases, the if clauses use the past subjunctive forms were and weren't, even if the subject is singular or the verb is present tense. The then clause--and this is usually a giveaway--will take the conditional form. This is usually indicated by the word would or one of the other past form modal verbs (could, might):
If I were a rich man, I wouldn't have to work hard. (But I am not a rich man.)
He could be much more successful if he weren't so negative. (But he is so negative.)
Let's turn, finally, to the sentence that your spelling checker flagged:
The letter claimed exactly the same as the first, namely that if his letter wasn't published, he would be angry.
Now, by the rules I have laid out you would expect that wasn't, the indicative form, would be appropriate because we didn't know if his letter was published. Actually, it is not what we know that matters but what the writer of the second letter knew, but it nets out to the same grammatical issue. If this were a counterfactual conditional, i.e., it was known that the letter wasn't published, then weren't, the past subjunctive form, would be correct. But it isn't a counterfactual, so wasn't is OK here.
So why would your spelling checker get it wrong? Because it looks so much like a counterfactual conditional. Remember my hint that the presence of would in the main clause is a giveaway that you have a counterfactual conditional. Well, it is except when it isn't, and this is one of those relatively rare cases where it isn't.
There is another grammatical pattern at play here--indirect quotes. Suppose that yesterday Bob said, and I quote, "I am late for class." I, today, quoting Bob indirectly, would say, "Bob said that he was late for class." Note two changes from Bob's statement to mine. First, Bob's first-person pronoun I becomes my third-person pronoun he. Second, present tense verb forms in Bob's statement become past tense verb forms in my statement, and the person changes from first to third, i.e., am becomes was. Note that all of the clauses in Bob's statement and mine use the indicative mood, present tense or past tense--no subjunctive or conditional mood clauses.
Now, let's do the same thing, but change what Bob said. Suppose that yesterday Bob said, "If my letter isn't published, I will be angry." And now, today, quoting Bob indirectly, I would say, "Bob said that if his letter wasn't published, he would be angry." Everything is indicative here. In particular, would is a past tense indicative form, not a conditional form, so in the associated if clause, the past tense indicative third-person singular form wasn't is used, not the past tense subjunctive form weren't. In this case, you are right, and the spelling checker is wrong. Now, hopefully, you know why.