Q. "Which bananas would you like?"
A1. "I'd rather have the yellow ones than the green ones."
A2. "The yellow ones rather than the green (ones).
Rather + than signals that a positive and negative preference are in apposition. But rather's location doesn't help you decide which one is which. It can be rather this than that or this rather than that where than signals the negative. The two have a slightly different aspect because rather tends to subordinate the clause it is attached to, so either the positive or the negative statement can be subordinated.
Q. "Do we really think anything will change after the election?"
A. "The contrary is rather to be supposed"
The answer unpacks as "You should rather suppose the contrary viewpoint than the stated one." It is a stilted way of suggesting that if history is any guide, you would be better off taking the other side of the bet. And being subordinated, it begs to be followed by some sort of justification. That is most likely it's true purpose - to be used as part of a rhetorical device which both establishes a (false?) dichotomy and prepares the audience for a particular argument.