There is no grammatical rule that defines the relative clause's antecedent here. Relative clauses can be placed in extraposition from the noun phrase it's modifying, meaning that they can be moved to the rightmost part of the sentence. Extraposition becomes more likely if the relative clause would introduce a substantive interruption to the flow of the sentence. So we can't rely on proximity alone to understand the relative clause.
In analyzing this sentence, there are two possibilities:
- The relative clause (that ran out to sea in vanishing flatness) modifies haze. "On the low shores" is a prepositional phrase that acts as an adverbial of place modifying rested. (Where did the haze rest? On the shores.) The extraposition of the relative clause avoids splitting the haze rested from on the low shores, which work together to describe the haze, its action, and its position. A logical progression is created from the haze's initial location to where it reaches. In other words, the haze (resting on the low shores) extends to the offing or the horizon.
- The relative clause modifies low shores. They go into the vanishing flatness - the shore into the sea, or perhaps the shore into the haze and the sea. While haze would remain the grammatical subject of the sentence, a lot of importance would be placed on the description of the shore.
Either is grammatically possible. Within the passage, this creates a blending or muddling effect on the horizon similar to that found at the start of this sentence shortly prior to the one we're discussing:
In the offing the sea and the sky were welded together without a joint ...
That already dabbles in literary interpretation beyond the standard for EL&U though, so I'll stop there.