It's like Mark Twain in Huck Finn and the like. She's writing phonically to convey how he sounds.
The yer in "tell yer that’s how it is at Hogwarts" just means "you" and is just a verbal affectation that has a long colorful tradition across the English speaking world like "idear" (for idea) or sawr (for saw) etc. The "intrusive r" it's called, I think. The main purpose of that one I presume is to convey Hagrid's uncivilized manner, talking like a "country bumpkin."
"If yeh think yer father’d rather you were expelled" is more of the same. The "yeh" being a common use of "you" especially in pirate (read: uncivilized) talk. The "yer" is just phonetic "your" but she wants you to read that he is distinctly saying "uncivilised" pronunciations. She's conveying that Hagrid would never pronounce it as "Yore." He means: "If you think your father would rather it be that you were expelled (instead of doing the "servant stuff") then get back to (notice "ter" the castle is just the author showing you his "intrusive r" again) the castle...
Edit: rereading yer inquiry, and noticing the second time how assuredly yeh confirm that you "know that yer means your..." I am now more sure that yer just not realizing that it is phonic writing. "Yer" does not definitively mean "your" any more than it definitively means "you're" or "you" for that matter. In this kind of writing (again, see Mark Twain, also Uncle Remus) the point is not a written grammar, but the sound, the accent that the character has.
(P.S. Not trying to be rude with my use of "yer" et al. Just having a bit of fun and demonstrating the style.)