It's an idiom. 

> Go to the devil<br>
> <sub>informal</sub><br>
>
>: used to forcefully and rudely tell someone to go away and leave one alone<br>
> // *She angrily told him to go to the devil.*
>
> ([Merriam Webster](https://www.merriam-webster.com/dictionary/go%20to%20the%20devil#:~:text=%E2%80%94used%20to%20forcefully%20and%20rudely,to%20go%20to%20the%20devil.))

"The devil with you" is just a shortened form of "To the devil with you!".