Starting with the facts of the matter, I find your version 3 to sound best, 2 almost as good, 1 and 4 rather awkward.
Turning to theory, Lawler's first comment is probably inspired by the treatment in McCawley's "Syntactic Phenomena" text, where the adverb "rudely" would be from a higher clause than the "speak to" clause -- something along the lines of "The manner in which [S Karen spoke to the manager] was rude" -- and the optional Passive Transformation applies or not to the embedded clause "[S Karen spoke to the manager]". Whether or not the Passive is applicable cannot depend on anything outside that clause, according to the cyclic assumption of TG, and so it can't depend on whether the manner adverb "rudely" is present.
But all that tells us is that the placement of "rudely" will not depend on whether a clause is a passive, but rather on the structure of "the manager was spoken to by Karen", whatever that structure is.
Now, "rudely" is a manner adverb which, McCawley argues, is a V-bar modifier, and the most natural place for any modifier is immediately before or immediately after the constituent it modifies (though there are many exceptions to this general principle). The passive has a V-bar "spoken to", and placing the adverb immediately before that gives us your version 3 (which sounds best, to me).
I don't know where the passive by-phrase fits into a passive sentence (this is discussed by McCawley), but if here the "by Karen" is a modifier of the V-bar "spoken to", that puts the adverb immediately before or immediately after "by Karen", which gives us your versions 2 and 4.
I don't see how to get your version 1.