It's an idiom (old-fashioned). 

> Go to the devil<br>
><sub>in British English</sub>
>
>: b. (interjection)
used to express annoyance with the person causing it
>
> ([Collins Dictionary](https://www.collinsdictionary.com/dictionary/english/go-to-the-devil))

Compare the structure of the phrase with the familiar "Off to bed with you!", which is a modification of the (established) phrase "[Off with you!](https://www.collinsdictionary.com/dictionary/english/off-with-you)". 

"Go to the devil" would become "Off to the Devil with you".

Similarly, "Go to hell" would become "Off to hell with you".

Boiled down even further, they would become:

- "[To] the devil with you"<br>
- "To hell with you"