It is only ambiguous when spoken. When read in text, there is a clear distinction between a case of a singular possessive noun [This boy's hat]; And the second case which is an example of a plural possessive adjectival-noun [This boys' hat]. [Boys'] is no longer the object being modified by [This] because it has taken the function of an adjective and is used, here, to describe what type of hat (a hat for boys as opposed to a hat for girls, for instance). Since we're using the plural of boys as our adjectival noun, which ends in the letter ess, we can't very well tack on an ['s] to indicate possession without making it look even more ridiculous -- [This boys's hat]. That just doesn't sound phonetically appealing -- so we just drop the second [s] and leave it at [s'] to indicate a plural noun (pretending to be an adjective) in its possessive form (also applies to singular nouns that happen to end with the letter [s] and their possessive forms).
I don't know that we call this type of ambiguity anything since, again, it's only phonetically ambiguous... Which I suppose would make it a
homophone - a word that is pronounced the same as another word but differs in meaning, and may differ in spelling.