Consider this example:
A few strong branches over water reach for what they ought not reach.
Which of the meanings comes closest to “ought not” in this sentence?
Is it “doesn't have to”, “should not” or “must not”?
Some might prefer the sentence to have to before reach, but otherwise it is well-formed. It’s impossible to be sure of the meaning out of context. It could be should not or it could be must not, or possibly something else, but not doesn't have to (which would in any case have to be don't have to in order to agree in number with the subject).
There appear to be several issues entangled here.
One issue, the answer to the OP's original question, is that they're all different modal constructions, each with different sources, different syntax, different idioms, and different possible senses, which nevertheless often overlap, in the right contexts. And they're all irregular as hell.
Another issue is the metaphoric source of ought, which is, as many of us know, the old past participle of the verb to owe, and is caught up with the concepts of credit and debit, authority and duty. One ought to do something if one owes one's action (to some person, or some abstract principle) to pay a debt, to pay one's due(s), to do one's due-ty, etc, etc.
Ought (or more commonly oughta) is principally a weak Necessity deontic modal, like should, rooted in a Face and Credit social system (rather than a Strict Authority system, like the strong Necessity modals must and hafta) but there is an epistemic sense as well, usually functioning wherever emotions can be invoked or inferred, as in anger over a misplaced item. Compare
which sounds a little weaker to my ear by comparison. But that could be personal usage bias. There's a lot of that, with all modals.
Still another issue is the use of ought as a true modal auxiliary verb (which means no to on infinitive complements), or as a semi-modal auxiliary, (like need or dare, which allow to before infinitive complements, under certain circumstances).
Ought is a paraphrase of a true modal auxiliary (in this case should) that hasn't quite assimilated yet, at least not enough to lose the to everywhere. Like the true modal auxiliary should, it precedes not, but there are variant past constructions -- hadn't oughta, shouldn't oughta. Also like should (and would, could, might and must), it's formed from a preterite stem but can refer to the future naturally.
But should never takes to. However, ought, if it's separated from its infinitive by a negative, can suffer stack overflow, lose its modal binding field, revert to regular verb status, and allow automatic generation of an infinitive complementizer to. This alternation of ought not go vs ought not to go resembles the behavior of need and dare, which are also Negative Polarity Items as modals, though with a different syntax.
Outside negative polarity environments, though, the to in ought to seems likely to remain for a while. It is reinforced by the bisyllabic nature of all the periphrastic modal proto-auxiliaries oughta, hafta, gonna, wanna, etc, which are all in the same kind of boat, but whose individual grammars all leak in different places.
In general usage, should not. "Doesn't Have to" does not really make sense to me in that context, whilst "must not" appears too strong.
A few strong branches over water reach for what they should not reach
However looking at that sentence it does seem a little odd. I am still confident that that is the best fit.
We normally say A few strong branches over water reach for what they ought not to reach.
I'd rephrase it as "... reach for what they aren't supposed to reach". The closest meaning to this is should not.
Furthermore, the meaning and the particular construction of the sentence is clearer when seen in its original context.
The last verse of the lyrics copied from the website; Always on the Run.
A few good men will go where they ought
Where they ought not be
And a few good mothers go for what they
What they ought not teach
And I long for a touch a reminder of us but
But it must not be
And a few strong branches over water reach for
What they ought not reach
My interpretation of the line (similar to @Irene's) is that the branches touch or reach for what they ought not to touch or reach. In other words, the protagonist should not reminisce over a relationship which has reached its conclusion.
Edit: In view of @tchrist comment below, the form: ought not + verb (bare infinitive) is commonly used in the US and is considered grammatically correct.
However, consulting my trusted Practical English Usage by Michael Swan, the accepted way to make negative infinitives in British English is by putting not before the infinitive. For further reference see this question "Infinitives with ought not" posted in July 2011.