I would say 'the black sheep of the family'.
According to Oxford Living Dictionary:
A member of a family or group who is regarded as a disgrace to it.
‘I'm happy enough to be the black sheep of the team, just like I'm the most downwardly mobile member of my family.’
‘As a young man, he was branded a black sheep after leaving his family's granite merchant business in Aberdeen and moving to Falkirk to launch his own firm.’
‘What we are is a family and like every family we have some black sheep and our behaviour is sometimes totally unacceptable.’
‘Every family has a black sheep, and I guess I was the one for our family.’
‘She is the black sheep of the family; drug rehab, shoplifting - you name it, she has done it.’
‘However, I am the black sheep in a family of shopaholics.’
‘Sarah is the free spirit black sheep of a rich family and is known for her impulsive, spontaneous personality.’
‘I was the black sheep of my family, getting in trouble trying to get rich.’
‘Having always been the extroverted black sheep in my family I'd always been very curious about these things we weren't allowed to do.’
‘Just suffice it to say that I have stood where you stand now - scorned by family members, labeled a black sheep.’
‘All families are untidy, with their unsolved mysteries, unspoken secrets, black sheep and messy relationships.’
‘Would my beliefs make me the black sheep of the family?’
‘Now, I'm the black sheep of the family, the ungrateful and neglectful daughter.'