You've described a general usage pattern that if 'impossible' is followed by a verb infinitive, then subject of the sentence (or clause) is 'it', but there are exceptions:

"In reality Strindberg was at this time almost impossible to live with." (he Road to Damascus, by August Strindberg)

"My uncle walked about the room in a state of excitement almost impossible to describe." (A Journey to the Centre of the Earth by Jules Verne)

"Even more basic questions regarding pain are impossible to answer:..." (Cyclopedia of Philosophy, by Sam Vaknin)

"There are “certain” words and conversations unhappily impossible to eradicate in schools." (The Brothers Karamazov by Fyodor Dostoyevsky)

"What followed was impossible to foresee." (The Road to Damascus, by August Strindberg)

So phrases '{Oranges are} {impossible to grow here}' and '{the lecturer is} {impossible to hear}' are probably acceptable in most contexts because they are not unclear.

But in general clarity would be lost by this construction because the 'impossibility' wouldn't clearly belong to the verb rather than the noun subject.

You've described a general usage pattern that if 'impossible' is followed by a verb infinitive, then subject of the sentence (or clause) is 'it', but there are exceptions:

"In reality Strindberg was at this time almost impossible to live with." (he Road to Damascus, by August Strindberg)

"My uncle walked about the room in a state of excitement almost impossible to describe." (A Journey to the Centre of the Earth by Jules Verne)

"Even more basic questions regarding pain are impossible to answer:..." (Cyclopedia of Philosophy, by Sam Vaknin)

"There are “certain” words and conversations unhappily impossible to eradicate in schools." (The Brothers Karamazov by Fyodor Dostoyevsky)

"What followed was impossible to foresee." (The Road to Damascus, by August Strindberg)

So phrases '{Oranges are} {impossible to grow here}' and '{the lecturer is} {impossible to hear}' are probably acceptable in most contexts because they are not unclear.

But in general clarity would be lost by this construction because the 'impossibility' wouldn't clearly be describing the verb. Something can be 'impossible' meaning stubborn, obstinate, etc. So it needs to be clear what is intended.