Here's a good explanation from mjv on Stack Overflow:
With a modal dialog, you set your application in a particular mode (a different "state" if you will), whereby only actions pertaining to that "mode" are accepted, hence preventing UI actions outside of the dialog.
At Andreas' prompting I thought I may have to dig dusty Windows API books, as often, the etymology/origin of a word or expression that has became broadly accepted is only found in early documentation, but in fact we still see this referenced in an online glossary from MS. The Modal entry reads (emphasis is mine):
modal
Restrictive or limited interaction due to operating in a mode. Modal often describes a secondary window that restricts a user's interaction with the owner window. See also: modeless.
Here's the full FOLDOC entry:
(Of an interface) Having modes. Modeless interfaces are generally considered to be superior
because the user does not have to remember
which mode he is in.
See modal logic.
In MS Windows programming, A window with the label "WS_MODAL" will stay on the screen and
claim all the user-input. Other windows can only
be accessed if the MODAL window is closed. Such
a window would typically be used for an error dialog box to warn the user for something important, like "Critical error, shut down the
system and restart".
The Jargon File (version 4.4.7) defines the noun mode:
[common] A general state, usually used with an
adjective describing the state. Use of the word
‘mode’ rather than ‘state’ implies that the state
is extended over time, and probably also that
some activity characteristic of that state is being
carried out. “No time to hack; I'm in thesis mode.” In its jargon sense, ‘mode’ is most often
attributed to people, though it is sometimes
applied to programs and inanimate objects. In
particular, see hack mode, day mode, night mode, demo mode, fireworks mode, and yoyo mode; also talk mode.
One also often hears the verbs enable and
disable used in connection with jargon modes.
Thus, for example, a sillier way of saying “I'm
going to crash” is “I'm going to enable crash
mode now”. One might also hear a request to
“disable flame mode, please”.
In a usage much closer to techspeak, a mode is
a special state that certain user interfaces must
pass into in order to perform certain functions.
For example, in order to insert characters into a
document in the Unix editor vi, one must type the “i” key, which invokes the “Insert”
command. The effect of this command is to put
vi into “insert mode”, in which typing the “i” key
has a quite different effect (to wit, it inserts an
“i” into the document). One must then hit
another special key, “ESC”, in order to leave “insert mode”. Nowadays, modeful interfaces
are generally considered losing but survive in quite a few widely used tools built in less
enlightened times.