Is the sentence:
Ann's friends and herself were really nice to us.
No, it is not grammatically correct. The pronoun herself requires there to be an antecedent, but there is none. Although the sentence mentions Ann's friends, Ann herself is not mentioned, so there is nothing for herself to properly refer to.
That is why the common formulation is Ann and her friends.
Separating the reflexive/intensive pronoun from its antecedent in that way sounds strange. The usual way you would phrase the sentence is:
Ann and her friends were really nice to us.
or, if you wanted to emphasize Ann:
Ann's friends—and Ann herself—were really nice to us.
Yes. It is grammatical.
Which is precisely what you wanted to know.
You may add, if you like: "If it is not grammatical, then what rules of grammar does it violate?" in your question.