Most of the references I can find about the word “ought” indicate that even when negating it, you should use an infinitive: “You ought not to go there.”
That sounds quite bad to my ear. Much better sounding to me is “You ought not go there.” Without the “to”.
Comparing this to similar usage of “should” does no good, since they can’t even agree without negation: “should go” versus “ought to go”.
Now that I think about it, I am suddenly far from certain that “ought to go” actually contains an infinitive in the first place; maybe “ought to” works together as some part of speech that modifies the verb, rather than “to go” being a unit.
I’m obviously no linguist. Can someone shed some light on this for me? Specifically, can you explain the function/part-of-speech of each word in a way that makes it clear whether "to go" or "go" is more correct?