It's an idiom. 

> Go to the devil<br>
> <sub>informal</sub><br>
>
>: used to forcefully and rudely tell someone to go away and leave one alone<br>
> // *She angrily told him to go to the devil.*

"The devil with you" is just a shortened form of "To the devil with you".