You ever nest two or more boxes together? That nest may have inspired later mathematicians and (eventually) everyone else to use nesting to suggest hierarchies of multiple nested things.
The Oxford English Dictionary has this definition for what I'll call the hierarchical nest (in "nest, v."):
- b. transitive. To place or fit (an object) inside another, esp. of the same kind; to place (an abstract element or entity) within the scope of another, esp. in a hierarchical structure; to arrange (abstract elements or entities) in a hierarchical conceptual structure. Usually in passive.
and this example:
1961 D. V. Huntsberger Elem. Statist. Inference ix. 230 Situations of this sort, where every classification is nested within the next larger one, are called nested or hierarchal classifications.
In the mid-twentieth century, as structuralist and systems-based thinking proliferated, hierarchical nest may have emerged, so that it applied to sets or hierarchies. For example, here is a 1943 mathematics paper that uses nested when working with sets (Wiener, Norbert, and Aurel Wintner. “The Discrete Chaos.” American Journal of Mathematics, vol. 65, no. 2, 1943, pp. 279–298):
Since m(E*) is independent of 1, it follows that in the sequence of successive decompositions of E* not more than m(E*) nested sequences of parallelepipeds contain points P. (p.288)
The mathematical usage extends even further back. In JSTOR, the earliest result for a search of "nested AND math*" rests in Ragsdale, V. “On the Arrangement of the Real Branches of Plane Algebraic Curves.” American Journal of Mathematics, vol. 28, no. 4, 1906, pp. 377–404. Ragsdale has to define the concept of nesting:
Hilbert proved that for n even, not more than (1/2)(n -2) of the p + 1 ovals can be nested; that is, so situated that the first lies inside a second, the second inside a third, and so on; and that curves of even order do exist having p + 1 ovals, (1/2)(n - 2) of which are nested; similarly, that for n odd, not more than (1/2)(n -3) ovals can be nested, if the maximum number of circuits is present, and that curves of odd order do exist having p + 1 circuits, (1/2)(n - 3) of which are nested ovals. (p. 378)
Ragsdale explains iterative nesting in order to emphasize how multiple, progressively smaller ovals can fit within the larger ones. In turn, this concept of nesting would possibly come from the practice of putting one container within another for storage or transport. The Annual Report - Iowa State Commerce Commission from 1889 contains several examples, including these items on the same page referring to baskets:
"baskets, not nested. in bdls ..."
"baskets, N.O.S., nested in bdls ..."
"Baskets, over-handled, covers and handles taken off and packed separately. Baskets nested in bundles."