First, the present participle *trying* does not act as a component of a progressive construction here; it acts as an adjective, meaning "annoying" or "tedious". 

> The lecture ... might have been tedious ... 

(That *trying* is not employed as a verb is evident from the fact that it is not followed by the complement which the verb *try* requires: either a frank NP, or a <i>to</i>-infinitive clause which acts as an NP.)

Consequently, the verb in the consequence clause (*then* clause, apodosis) of this conditional sentence is simply *might have been*: the ordinary past irrealis form for *MAY BE*.

The condition clause (*if* clause, protasis) is expressed without *if* by subject/auxiliary inversion.

> ... if the professor<sup>†</sup> had not leavened ... → ... had the professor not leavened ... 

The verb in this clause is, again, a past irrealis, expressed with a past perfect.  

You may perhaps see the structure more clearly if we restore it to canonical *if ... then* order:

> If the professor had not leavened the lecture ... [then] it might have been trying ...
 
The irrealis forms tell us that the condition (*had he **not** leavened*) did not occur—rather, the professor **did** leaven his lecture—and that as a result the consequence which actualization of the condition would have triggered (*might have been trying*) did not occur either.

The sentence thus signifies that the professor leavened his lecture with humorous asides, which prevented it from being trying for nonspecialists in the audience.

I cannot understand your question about using *not* at the end of the sentence; if I have not resolved it, please let me know what troubles you about the *not* and I will endeavour to satisfy you.

***
<sup>†</sup> I take it that your bare *professor* is a typo; I have arbitrarily read it as *the professor* rather than *Professor X*.