The most likely answer to the first question is that one would simply refer to the facial hair above the upper lip as a kind of **beard**.

An early attestation in *OED* under "mustachio" from **1551** -- in fact, the earliest attestation of either "mustachio," "mustache," or any variants, explicitly denotes the word this way.

>**1551** They suffer their **mostacchi** to growe a quarter of a yarde longer than their beardes. [margin] **Mostacchi is the berde of the vpper lyppe**.

- William Thomas · *G. Barbaro's Trav. Persia* 

"Beard" is attested as early as *circa* 825, making it a much earlier term than "mustache."

Using the word "beard" to refer to a mustache didn't disappear either. This citation from 1760 poses an example of such a use, even though the word mustache would have been well-established by the time of its writing.

>**1760**: [The Britons] shaved the **beard** on the chin; **that on the upper lip** was suffered to remain.

- Edmund Burke · *An essay towards an abridgment of the English history* · 1760.

----------

As to why the word "mustache" made its rise in the 16th century, I can only speculate.  I have no reason to believe the word grew out of any particularly notable use, like that of Shakespeare in **1598**.

> It will please his Grace..sometime to..dallie with my excrement, with my **mustachie** 

- ["Love's Labor Lost" from *The Complete Works of William Shakespeare*][1]

It would be another notch on a very long belt.


  [1]: https://books.google.com/books?id=xVSWAAAAQBAJ&pg=PT2993&dq=It%20will%20please%20his%20Grace..sometime%20to..dallie%20with%20my%20excrement,%20with%20my%20mustache&hl=en&sa=X&ved=0ahUKEwjwlfabiezVAhVoyoMKHYNtCaAQ6AEIKDAA#v=onepage&q=It%20will%20please%20his%20Grace..sometime%20to..dallie%20with%20my%20excrement%2C%20with%20my%20mustache&f=false