The word existed, and exists, with a different meaning in geology, in which it refers to crystals trapped between other rocks and crystals (I may have the geological details of that not quite right). It is used in such works as Edward Huntington Williams' Manual of Lithology: Treating of the Principles of the Science with Special Reference to Megascopic Analysis ... from 1895, which considerably predates the Alien franchise.
In both this case and the films, the etymology is from the establish process of combining Latin and/or Greek words (in this case both Greek) and so ξένος (stranger, foreign, alien) and μορφή (shape, form, body).
It applies in the geological sense, because such crystals are foreign bodies within another rock.
It applies in the film because such creatures are alien/foreign to all organisms otherwise known to the protagonists.
(Incidentally, as used in the film it does not refer specifically to the eponymous aliens; it refers to any real or hypothetical non-terrestrial being. If, on their way to LV-426, the crew of the USS Sulaco came across a being that looked like a lovable turd, asked to "phone home" and healed people with a glowing finger while saying "ouch", they would have called that a xenomorph too [and then cut him up to see if his finger was of use to medical research]).
Both the Greek words involved have a history of being used in English, such as xenophobe and xenoblast for ξένος and endomorph and lagomorph for μορφή. This last example compares well, as it is similarly used to describe creatues (it covers both rabbits and hares, and literally means "having the form of a hare").
As such, it would be a pretty unsurprising coinage for a scientist to make, whether a real 19th century scientist looking at crystals in rocks, or a fictional 22nd century scientist describing an extra-terrestrial species.