The decision of which to use is determined by context. Usually, the surrounding text (or spoken word) would clue the audience into what kind of door you're talking about, so you could just say "door."
In American English, we would probably say "front door" or "exterior door" to refer to the type of door (i think that) you're referring to.
But as for your examples, "doors of the houses" sounds like it would refer to the doors of separate houses. For example, "The doors of the houses on my street all had Christmas wreaths on them."
I don't think that "house door" is a very common phrase, but I don't see anything wrong with it grammatically. If you were to say "house doors," it could refer to a house with multiple doors. In that sense, though, "house's doors" might be more common. I suppose it could also refer to several doors that were built to be installed in houses. "These house doors must be delivered tomorrow." But again, if context doesn't indicate the type of door, you'd probably say something like "exterior doors" or "closet doors."