Let's try some minimal pairs:
- I walk rather than run
- I walk instead of running
- I eat apples rather than oranges
- I eat apples instead of oranges
- I walk quickly instead of quietly
- I walk quickly rather than quietly
I don't really think there's much of a difference, except perhaps "rather than" implies preference whereas "instead of" implies substitution. But that might be nitpicking. And it doesn't seem to apply when using it to coordinate adverbs (last pair).
Also, there's a slight difference in the verb forms in the first pair, but there's not really a difference in meaning there.