Even if there is a rule saying that 'her' is right here, for me there's a big problem with 'her' - it makes it sound like there is only one lesson, that was assigned to the sister.
Suppose Mohan and his sister were trying to open the door or doors of a car:
Neither Mohan nor his sister could open his door
Both are trying to open the door next to Mohan's seat
Neither Mohan nor his sister could open her door
Both are trying to open the door next to the sister's seat
Neither Mohan nor his sister could open their door
The two people are trying to open two different doors.
I can't see any way of using 'her' in your sentence while also being consist with the sense of there being more than one lesson.