TlDr: I don't think we know exactly but most likely the fact that a white house is a thing makes it important that we use the White House. At the same time Blair House even though it's head is not a proper noun is special enough that we omit the.
This is an interesting difference which depends on the difference between weak and strong proper names. Wherever I'm quoting in the following it's from Huddleston and Pullum's CGEL. (and wherever I'm not quoting it's also from CGEL but rephrased.:) )
Both the White House and Blair House are proper names1. Proper names are
expressions which have been conventionally adopted as the name of a
particular entity - or, in the case of plurals like the Hebrides, a
collection of entities.
All proper names refer to a specific entity and the entity is inherently definite. That means you can't use "a" with them, and don't need to use "the". But
We distinguish, then, between strong proper names like Kim and New York, where there is no determiner, and weak proper names like the Thames or the Bronx, where definiteness is redundantly marked by the definite article the.
It turns out that the White House is a weak proper name unlike Blair House which is strong. That doesn't actually explain much though since the next obvious question would why is one weak and one strong. The answer to that I couldn't actually find.
It turns out there are some rules regarding weak and strong proper names. Namely plural proper names are always weak. Mountain ranges and island groups are examples of these. Countries usually have strong names though there are exceptions, the United Kingdom, the United States of America, the Netherlands. Usually weak proper names with proper nouns as heads include: geographically defined regions, famous buildings, rivers, straits, seas, oceans, deserts, political/military authorities, libraries, galleries,museums,religious tracts.
On the other hand names of individual mountains are usually strong (Ben Nevis, Mount Everest, Nanga Parbat) "but those in the Swiss alps may have weak names" (the Eiger, the Jungfrau, the Matterhorn).
My personal opinion regarding why there it is the White House as compared to Blair House is that Blair in this case is a proper adjective. The same thing happens with places like Hatfield House, Chatsworth house, Burghley House and really just about any named house in England whose name comes from the name of a family or town.
1: These are not the same as proper nouns "which are word-level units belonging to the category noun, Clinton and Zealand are proper nouns, but New Zealand is not."