If the facial hair is very soft and has a slightly matted appearance it is sometimes referred to as down. Some compare it to the skin that covers a peach. It is also used for very young children, especially newborns.
down noun
2.1. Fine, soft hair on the face or body of a person:
the little girl had a covering of golden down on her head
Oxford Dictionaries
For facial hair that is visible above the lips, there are two ways of calling it; (fine) upper-lip hair, which many would posit is an euphemism for the more masculine term, moustache. Female moustaches aren't the same as men's. Firstly they don't normally grow beyond a certain length, and secondly they're usually much softer and more delicate in appearance.
If you type upper-lip hair on Google you'll find many instances and images
moustache noun
1. A strip of hair left to grow above the upper lip.
She has a pencil-thin moustache on her upper lip
Oxford Dictionaries
A more precise medical term for facial and body hair is
Vellus hair
Vellus hair is short, fine, light-colored, and barely noticeable hair
that develops on most of a person's body during childhood. Exceptions
include the lips, the back of the ear, the palm of the hand, the sole
of the foot, some external genital areas, the navel and scar tissue. [...]Vellus hair is most easily observed on children and adult women, as they generally have less terminal hair to obscure it.
Wikipedia
For hair that is visible around the chin area, you could call it chin hair, but I barely read this term in depilatory products. Generally the most common and most understood term is the one discovered by the OP himself, facial hair.