There is what I believe to be a literary device that describes recursion, particularly in footnotes. For example, a footnote of a footnote. You can also have a footnote, that is the footnote of a footnote, and it can go as deep as you want it to. This would be known as ____ in literature. Recursion is taking place, but it's not the word I'm looking for. I believe I discovered this effect when looking at a list of literary devices but my Chrome history just isn't helping me find it again!
There is what I believe to be a literary device that describes recursion
Mise en abyme is originally a heraldic term for a coat of arms put in the centre (en abyme) of another coat of arms. In terms of literature it is used of a story or related incident that is relayed as part of another story, and reflects that larger story (the play Hamlet has produced in Hamlet is perhaps the most frequently cited case). Some examples go so far as to have a mise en abyme within a mise en abyme, possibly even truly recursive in that it goes back on itself (e.g Gaiman's comic-book story Worlds' End is a story about a group of travellers in an inn "at the end of the world" telling stories, which give insight into the wider series that Worlds' End was part of. One of those stories involves a night where a teacher is telling pupils stories, including one about travellers in an inn "at the end of the world" telling stories).
Mise en abyme is also used in terms of other artforms, and the visual arts more readily allow for the perception that there are indeed infinite recursive mise en abyme within the picture. Literature can hint at that (as per the Gaiman story I mentioned above) but not as easily. This is sometimes called the Droste effect after Droste cacao which came in a tin with a picture of someone holding a tin of Droste cacoa, with a picture of someone holding a tin of Droste cacoa, and so on.
For example, a footnote of a footnote. You can also have a footnote, that is the footnote of a footnote, and it can go as deep as you want it to.
That's a nested footnote. To be recursive it would have to relate directly to itself1, or follow a circular path2, and that we would just call that recursive footnotes or circular footnotes. It's not a very common technique so not as widely used as mise en abyme though it can of course overlap with mise en abyme or for that matter so can just the use of footnotes at all; some of Pratchett's footnotes in his comic Discworld series can be considered as very brief mise en abyme in how they relate to the wider story.
See footnote 1.
This would require a lot of footnotes4
Footnotes being rare in literary use generally5, we might consider just about any use of footnotes in fiction to be "a lot".
Of course a lot is relative3.
Though some authors make heavy use such as David Foster Wallace using footnotes6 allowing for a straight read but also a reading approach that forces a lot of back-and-forth between different frames2, and Susanna Clarke using them to give an impression of the work being a non-fictional account.
Strictly speaking, Wallace used endnotes, not footnotes in his most famous work.