What does the expression "The devil with you" mean in this paragraph?
“Yes, yes, I know all about it. Your dear sainted mother is the only woman you’ll ever let into your heart, more’s the pity. Let me tell you, boy: yes, I loved your mother, in the end, but it didn’t start out like that. It was a good match from the beginning, a smart and practical one. That’s all one can hope for when securing a legacy. And I’ll be damned if I go to my grave knowing…” Lord Berkeley blustered, and then fell into a fit of coughing, which ended whatever impassioned speech he was about to give. He clicked his fingers, and a servant rushed over with a glass of water. Eventually, he calmed himself, his fit of pique subsiding, and he waved a hand at his son. “The devil with you,” he said, quietly. “Maybe when I’m gone, you can ask this one here all the questions you never asked me in life.”