You might call it your household:
The inhabitants of a house considered collectively; a group of people (esp. a family) living together as a unit; a domestic establishment (including any servants, attendants, etc.).
(Oxford English Dictionary)
However, this of course presumes that your family is not separated. It does sound a little archaic though, but saying something like:
The rest of my family is very conservative but my household is quite liberal.
The definition states it includes servants, but I wouldn't jump to this understanding as an American, it might be different in other countries or social levels.
There's also family unit:
n. a family considered as a single entity, esp. as a discrete constituent of society; the primary social group, traditionally consisting of parents and their children.
But this sounds almost statistical in nature to me, I would be hard-pressed to use it naturally in conversation.