Is there a single word for when a 'nerd' or expert talks so in depth about a topic that it leaves everyone else behind? I have looked in the reverse dictionary and cannot find one. A phrase could be 'they talked over every one's head'.
I suggest technobabble:
Technobabble (a portmanteau of technology and babble), also called technospeak, is a form of jargon that uses buzzwords, esoteric language, specialized technical terms, or technical slang that is incomprehensible to the listener.
In a sample sentence:
His presentation was technobabble to most of the audience.
@Scot proposal "Jargon" is good.
Rigmarole (or rigamarole) is also a possibility.
Definition: confused or meaningless talk; a complex and sometimes ritualistic procedure.
Examples: We had to go through the rigmarole of installing, registering, and activating the software before we found out it wouldn't work. He just told us what to do without all the usual rigamarole.
Gibberish fits the bill:
unintelligible or meaningless language:
a : a technical or esoteric language
b : pretentious or needlessly obscure language (MW)
For example: his writings are nothing but gibberish to me.
You might describe his speech as baffling, incomprehensible, indecipherable, inaccessible or impenetrable. I particularly think "impenetrable" or perhaps "inaccessible" suits your purposes, since either clearly suggests that the contents of the speech were pitched at too high a level for the group to understand. (A slight problem with "incomprehensible" or "indecipherable" is that it leaves open the possibility the audience didn't understand because the talk was, in fact, nonsense, rather than it just being difficult material.)
An informal alternative is highfalutin or highfaluting.
(Especially of speech, writing, or ideas) pompous or pretentious: you don’t want any highfalutin jargon — Oxford Dictionaries
But that suggests the writing style was overblown and pompous, and the speaker was deliberately trying to make himself look clever in front of the audience. It might be that your speaker was just making a very dry, detailed and possibly technical speech and, though this was not the intended effect, the audience were lost by it.
You say "expert" but it isn't necessarily clear that it is in a technical field. If so those suggestions "technobabble" or "jargon" work well. In other domains, wonkspeak or wonk-speak might be appropriate, particularly in areas like educational theory or public policy. Politicians are often criticised for overuse of "wonkspeak", see e.g. "Jennifer Granholm: Biden and Ryan will have to watch the wonk-speak" in the Washington Post, or "The trouble with "Wonkspeak" in Prospect Magazine:
GDP is the single economic metric that can be translated from Westminster Wonkspeak into language digestible in the outside world and despite its shortcomings, it is the most significant political economic number that exists.
"Westminster" is the home of the British parliament; for the American equivalent see e.g.
Health care reform is insanely complicated, as opponents love to point out in lavishly detailed flow charts and spaghetti-shaped graphics detailing the various proposals. The issue is extremely important to the American people, but the policy details turn them off. If the president veers off into Washington wonk-speak – as he is sometimes inclined to do — he’ll drive viewers away. — Eamon Javers, "Seven things to watch tonight", Politico (9 September 2009)
This sounds like gobbledygook.
language that is meaningless or is made unintelligible by excessive use of abstruse technical terms; nonsense.
There is also some interesting information on the origin of that word in the source I linked to.