In my neighborhood, most yards have two houses: The major residence where the family lives (usually in the front), and then another house in the back. I thought the second house was called a "guest house" but that does not seem to be the case. When I lookup "guest house" the results do not capture the idea that the second house is smaller. So what is the proper name for those?
Usually guest house. Although at my house we simply called it "the little house".
From the Oxford Dictionary: Guest House - A small, separate house on the grounds of a larger one, used for accommodating guests.
In British English a guest house is more often something like a small hotel, though it may refer to accommodation for visitors to a particularly grand property. Instead the word "annexe" (also "annex") may be used, defined as "a subsidiary building or an addition to a building" (dictionary.com).
The reason behind this (speculation) may be that plots of land for normal houses tend to be smaller in the UK than many countries. To build such a subsidiary house is less common and uses a large proportion of the plot. This means when it exists it's likely to be smaller and may well abut the main house (though not always, I've seen them converted from detached garages for example)
In the UK, if these buildings are attached to the main hoouse then they are called adjoining houses or adjoining buildings.
For instance, some houses here in the UK have a small cottage alongside a main house, and this is called the adjoining cottage.
Adjoining: (Of a building, room, or piece of land) next to or joined with