Let's try some minimal pairs:
I walk rather than run
I walk instead of running
I eat apples rather than oranges
I eat apples instead of oranges
I walk quickly instead of quietly
I walk quickly rather than quietly
I don't really think there's much of a difference, except perhaps
rather than implies preference whereas
instead of implies substitution. But that might be nitpicking. And it doesn't seem to apply when using it to coordinate adverbs (last pair).
Also, there's a slight difference in the verb forms in the first pair, but there's not really a difference in meaning there.