I believe your question specifically is, "Does otherwise carry an implication of undesirability?"
Let's use a simple example (set in a restaurant):
"I'll take a Coca-Cola. Otherwise, I'll take a Pepsi."
I think in this case, it is clear that the speaker prefers the first option, i.e., the Coca-Cola, over the option that follows otherwise. So, it is appropriate to conclude otherwise carries a negative connotation?
To the dictionary!
Otherwise, as an adverb, has three meanings:
1. In circumstances different from those present or considered; or else.
Example: I’m not motivated by money, otherwise I would have quit.
2. In other respects; apart from that.
Example: There was an otherwise totally black cat with a single white whisker
3. In a different way.
Example: "He means mischief—it’s no good pretending otherwise."
3. 1 As an alternative.
Example: His name was Leonardo Pisano, otherwise known as Fibonacci.
The issue with otherwise is it means both or and if not. Or is a neutral term whereas if not implies the previous option is preferred.
In your example, you mitigate the ambiguity by prefacing the sentence with, "If you would like it." This phrase shifts the intent away from the writer and to the reader, thereby indicating that the writer does not necessarily prefer the first option.
If your goal is to make it appear that you, the writer, have no preference, then I believe your current sentence achieves this (as long as you leave the introductory clause in there).
Though, if you wanted to remove any element of uncertainty, you could always forget the pesky otherwise altogether:
"I can come along with you to your house or follow the address. Your pick."
Otherwise, I think you are okay. ;)
(Oxford Dictionaries Online)