First, the present participle trying does not act as a component of a progressive construction here; it acts as an adjective, meaning "annoying" or "tedious".
The lecture ... might have been tedious ...
(That trying is not employed as a verb is evident from two facts:
- It is not followed by the complement which the verb try requires: either a frank NP, or a to-infinitive clause which acts as an NP.
- The adverb particularly cannot fall before the participle in a present perfect progressive. )
Consequently, the verb in the consequence clause (then clause, apodosis) of this conditional sentence is simply might have been: the ordinary past irrealis form for MAY BE.
The condition clause (if clause, protasis) is expressed without if by subject/auxiliary inversion.
... if the professor† had not leavened ... → ... had the professor not leavened ...
The verb in this clause is, again, a past irrealis, expressed with a past perfect.
You may perhaps see the structure more clearly if we restore it to canonical if ... then order:
If the professor had not leavened the lecture ... [then] it might have been trying ...
The irrealis forms tell us that the condition (had he not leavened) did not occur—rather, the professor did leaven his lecture—and that as a result the consequence which actualization of the condition would have triggered (might have been trying) did not occur either.
The sentence thus signifies that the professor leavened his lecture with humorous asides, which prevented it from being trying for nonspecialists in the audience.
I cannot understand your question about using not at the end of the sentence; if I have not resolved it, please let me know what troubles you about the not and I will endeavour to satisfy you.
† I take it that your bare professor is a typo; I have arbitrarily read it as the professor rather than Professor X.