From the paper Dialectal Variation in Number Agreement with Collective Nouns in English, by
Ingrid Rodrick Beiler and Cynthia Hatch [talking about collective nouns]:
Levin (2001) concurs: “Spontaneously produced AmE speech appears to contain high proportions of plural agreement with relative and personal pronouns, whereas more formal AmE preserves low proportions of plural agreement...Verbs, on the other hand, very rarely take plural agreement in AmE” (p. 76).
That is, in informal American English, we rarely use plural verbs for nouns like family and crowd, but we are happy using plural pronouns for them. In formal American English, we are more likely to avoid using plural pronouns. But in formal writing, we have the luxury of having time to think about how to rewrite the sentence to circumvent the problem.
Levin also studied mixed agreement, where the verb is singular and the pronoun is plural:
... results from the spoken AmE corpora indicate that the collective nouns, committee, company, family, and group produce shifts “in at least two thirds of the instances” (Levin, 2001, p. 120).
So sentences like the OP's,
The group of students does their work well.
The family returns to their house.
are quite common in informal American English. It's probably a good idea to avoid them in formal writing—although you certainly don't want to use its in sentences like
The family put on their coats.
You should say the family members, or figure out some other way of paraphrasing the sentence. On the other hand, meaning matters:
The family holds its annual reunion in Annapolis.
is perfectly fine.
1 Levin, M. (2001). Agreement with Collective Nouns in English. Lund, Sweden: Lund University.