You are correct that where there is a direct object, adverbs of manner are most commonly inserted after it, and moving them to other positions can render the sentence unnatural or ambiguous.
The agency enforced the new rule sporadically at best.
*The agency enforced sporadically at best the new rule.
This is not a strict rule, however, especially in writing or more formal communication. You can adjust the position of the adverb to fit your needs, though it is hard for me to distill acceptable occasions to a rule of thumb. Here are a few examples.
When the object is longer than a few words, moving the adverb to a medial position more clearly establishes that it modifies the main verb.
From Franklin Roosevelt's first inaugural address:
[W]e must frankly recognize the overbalance of population in our industrial centers…
This could be rendered We must recognize the overbalance of population in our industrial centers frankly, but that increases the cognitive load on the listener to associate frankly with must recognize, whereas must frankly recognize (or frankly must recognize or must recognize frankly) does not.
Moving the adverb closer to the verb has the effect of emphasizing it, which can be desirable for rhetorical effect. From the same speech:
I assume unhesitatingly the leadership of this great army…
In this dedication of a Nation we humbly ask the blessing of God.
Moving the adverb in front of the objects can make parallel constructions more clear, or draw attention to them, as in this famous syllepsis:
He was alternately cudgeling his brains and his donkey.