What does the expression "The devil with you" mean in this paragraph?
“Yes, yes, I know all about it. Your dear sainted mother is the only woman you’ll ever let into your heart, more’s the pity. Let me tell you, boy: yes, I loved your mother, in the end, but it didn’t start out like that. It was a good match from the beginning, a smart and practical one. That’s all one can hope for when securing a legacy. And I’ll be damned if I go to my grave knowing…” Lord Berkeley blustered, and then fell into a fit of coughing, which ended whatever impassioned speech he was about to give. He clicked his fingers, and a servant rushed over with a glass of water. Eventually, he calmed himself, his fit of pique subsiding, and he waved a hand at his son. “The devil with you,” he said, quietly. “Maybe when I’m gone, you can ask this one here all the questions you never asked me in life.”
(Sherlock Holmes - The Red Tower by Mark A. Latham)