It's an idiom. > Go to the devil<br> > <sub>informal</sub><br> > >: used to forcefully and rudely tell someone to go away and leave one alone<br> > // *She angrily told him to go to the devil.* > > ([Merriam Webster](https://www.merriam-webster.com/dictionary/go%20to%20the%20devil#:~:text=%E2%80%94used%20to%20forcefully%20and%20rudely,to%20go%20to%20the%20devil.)) "The devil with you" is just a shortened form of "To the devil with you!".