I understand that typically in American English, farther is used for physical distance while further is used for metaphorical distance (i.e. farther to the restaurant vs. further in the book).
However, which word would you use when using cardinal directions? You need to head further north or farther north?
You can use either farther or further in reference to cardinal directions, so, using your example, both "farther North" and "further North" are correct in usage and mean the same thing.
Here's what you said:
"I understand that typically in American English, farther is used for physical distance while further is used for metaphorical distance."
Therein lies your problem. That's not correct.
What would be correct is:
"...farther, by definition, is used for literal distance, while further, by definition, is used for both literal distance and figurative distance."
Anytime you use farther, you can aptly use further instead. That's because one definition of further is farther, meaning further can be used for literal distance, just like farther can. However, the converse is not true. Since farther doesn't expressly include using it for figurative distance among its definitions, farther can't be used everywhere further is used.
Now, that doesn't mean that farther can't ever be used in a metaphor or used figuratively. It's no different than any word and can be. It's just that farther's dictionary definition only includes using it for literal distance— or "physical distance," as you put it. Meanwhile, further's literal definition does include using it for figurative distance.
EXAMPLE:
A great example of this distinction between farther and further is found in Ford's ad slogan "Go further!" I laugh out loud every time I hear or see those Ford ads. To anyone accutely aware of the difference between further and farther, it's blatantly obvious Ford is trying to pull a fast one by not using the obviously more relevant choice farther but instead using the more ambiguous choice further.
You see, in these ads, Ford is overtly advertising reliability and longevity, clearly conveying through accompanying context and imagery that Ford's vehicles outlast its competitors' vehicles, going more total miles to achieve higher ultimate mileage than its competitors' vehicles. So Ford is conveying and clearly wants customers to infer that Ford vehicles "Go farther!"
However, Ford's ad slogan quite conspicuously isn't "Go farther!" It's "Go further!" Why?
It's because further has the added definition of figurative distance, so by using further, Ford implies further's definition of farther but can fall back on further's figurative definition to avoid any civil liability for false advertisement and civil liability for any expressed warranty the ads may create should anyone sue Ford after their Ford didn't do what Ford wants customers to infer and fails to literally go a greater distance than Ford's competitors' vehicles.
Even though Ford bends over backwards in these ads to get customers to infer the meaning farther from further, using the word further instead of farther gives Ford the legal wiggle room to defend itself in any lawsuit that it wasn't speaking literally but merely figuratively, that it was, for example, saying the plaintiff would figuratively "go further" in life by driving a Ford, a subjective, opinion-based, legally unactionable measure, not literally go more miles in a Ford vehicle than in a competitor's vehicle, not "Go farther!" Ford could then further underscore that it didn't mean literally more miles by claiming that further's figurative meaning was precisely the reason it chose "further" over "farther" for the ad campaign.
