The Benjamin of the family is an idiom / biblical reference sometimes used in this sense. From Wikipedia:
The "Benjamin of the family" is a phrase used in several languages to refer to the youngest son – especially when he is much younger than his brothers. [...] Sometimes the name is chosen for a son born to mature parents unlikely to have more children, especially if he has several older siblings. Both of these usages derive from the biblical son of Jacob of that name, who occupied that position in his family.
I've never actually heard this idiom uttered aloud; it might be dated or rare. However, it's on point!