It's an idiom.
Go to the devil
in British English: b. (interjection) used to express annoyance with the person causing it
Compare the structure of the phrase with the familiar "Off to bed with you!".
"Go to the devil" would become "[To] the Devil with you".
Similarly, "Go to hell" would become "To hell with you".