We can use preposition phrases to postmodify noun phrases.
In the Original Poster's example, the verb find takes a Locative Complement the essential core of which is in a layer. This Locative Complement explains the location of the oxygen.
Now within this preposition phrase there is the noun phrase a layer. The Head noun in this noun phrase is being postmodified by another preposition phrase in the earth's atmosphere.
This preposition phrase is in the earth's atmosphere is being modified by the 'flat' adverb high. So the full preposition phrase here is high in the earth's atmosphere. The word high belongs with in the atmosphere, not with the noun layer
The structure of the Locative Complement therefore is:
- in [a layer [high in the earth's atmosphere]]
We can paraphrase the sentence like this:
- a form of oxygen that is found in a layer which is high in the earth's atmosphere.
Here is a sentence with a Locative Complement with the same structure:
- We went to a location high in the mountains.