Depending on the context, it can refer to many things. For example, here is an instance of the collocation "keep it together" from Edward Bellamy, Miss Luddington's Sister, 1884, as found in the Corpus of Historical American English. "It" is a dress that is falling to pieces:
I forgot to say that the reason the dress all fell to pieces the day after I came here was that it had been treated with a chemical preparation, which had completely rotted the texture of the cloth. Indeed I had trouble to keep it together that first night.
In other contexts, it is more ambiguous, but refers to a kind of order. Here is an excerpt from a letter from George Washington to General Schuyler in 1779, here reprinted in 1809:
Since the date of my last, we have had the virtue and patience of the army put to the severest trial. Sometimes it has been five or six days together without bread; at other times, as many days without meat; and once or twice, two or three days, without either. I hardly thought it possible at one period, that we should be able to keep it together.
The idiomatic potential of the collocation is clear at least a century earlier, where contextual uses suggest that keeping it together (whatever it is) was better than the alternative. For example, an armory (a description of coats of arms) called The academy of armory by Randle Holme (1688) describes one coat of arms where a hill is held together by briars; keeping it together (solidarity) is the message:
He beareth Argent, a Rock Gules, piled at the foundation and Enwraped with Osiers, Bri∣ers, or Thornes, Tenne. This we must suppose to be a kind of Sandy Hill, or Mountain; which would in time moulder & fall away, if it were not defended, & support∣ed, or fortified with stakes, and windings to keep it together.
A theological text, A discourse concerning the divine providence, by William Sherlock (1694), contrasts the world staying together with the world falling apart, with keeping together an obvious good:
for a World that came together by Chance, and has nothing to keep it together but the Chance that made it, which is as uncertain and mutable as Chance is, will quickly unmake it self.
So even before 1800, keep it together could refer to maintaining order. More modern uses referring to emotional states may be as old as the 1970s, as this tantalyzing bibliographic entry suggests:
Jenkins, Flo. "How the O'Jays Keep It Together." Right On! 4, no.1 (November 1974): 26-27.
And there's also the 1970 song by Paul Davis, "I Just Wanna Keep It Together," referring to a broken relationship.
What is it in colloquial usage today? Green's Dictionary of Slang explains that it refers to emotional control or appearances. The phrase has been generalized so that one need not explain exactly what it is. It's a dummy-it standing in for anything from one's mental state to one's routine to one's appearance of being in control. You could keep your cool, keep your shit/jazz/jive together, keep your stuff together, or keep your wits together. All of those usages are clustered within a few years - it could have come from any or all of them, as the second excerpt (Black America by John F. Szwed, 1970) models:
Of high value then is getting and keeping your stuff or shit together, which means knowing at all times what you're doing and being able to use your best verbal devices. Shit as used here is synonymous with jazz and jive and stuff, simply meaning good talk. Cool then is keeping your jazz together, remaining well-ordered ...
Today the usage has moved far away from it referring to any single thing, which is why it's called a dummy.