It's an idiom (old-fashioned):
Go to the devil
in British English: b. (interjection) used to express annoyance with the person causing it
Compare the structure of the phrase with the familiar "Off to bed with you!", which is a modification of the (established) phrase "Off with you!".
"Go to the devil" would become "Off to the Devil with you".
Similarly, "Go to hell" would become "Off to hell with you".
Boiling them down would result in:
- "[To] the devil with you"
- "To hell with you"