Early instances of 'reality check' as a noun phrase
The earliest Google Books match I was able to confirm for "reality check" as a noun phrase is from "The Case of Gregor: Interpretation of Test Data," in Symposium presented at American Psychological Association Meeting, Denver, 1949 (1949) [combined snippets]:
Contaminatory thinking, also based partly on diffuseness, is seen in his saying that dog and lion are similar in that they are "animals, wild animals," where the concept of wildness presumably spreads from lion to dog without any subsequent reality check. Impaired communicative facility is evident in his saying that the similarity between wood and alcohol is that wood is in alcohol, a verbalization which makes no immediate sense, which requires further elaboration, but which he lets go at that.
Another early match appears in Bernard Notcutt, The Psychology of Personality (1953) [combined snippets]:
This begins as a realistic identification, and develops into a wish-fulfilment fantasy, in which the reality-check lapses. We notice how he alters the story as he goes along. At the beginning the violin is his, but a little bit later he would like to have one for himself. As he slides into the identification he remembers that there are a lot of things he would like to have, of which he feels deprived (such as a home or a wife of his own perhaps).
And a third is from Samuel Beck, The Six Schizophrenias: Reaction Patterns in Children and Adults (1954) [combined snippets]:
Patient can't put the card away. He has his way with the person giving him the Rorschach. You take page after page of response, and you are stuck. The thing structurally is that there is a failure of the ego to grow, although there is this interest in overt productivity. Secondly, the great amount of M, eight fantasy responses, which is larger than before. There is no change in reality check. The ego is the same, so we have to say that he now day-dreams without attending to reality. Therefore he is more schizoph[re]nic now than he was at the time of his first Rorschach, because of his autistic solutions.
The obvious common element here is that each instance arises in the field of psychiatry and appears to use "reality check" in the sense of a comparison of a fanciful description to a more sober and plausible fact-based description. This likewise appears to be the setting in which the OED's first citation of "reality check" (from Murchison's Handbook of Social Psychology, 1935), as cited in a comment by weissj above, uses the term. Google Books does not have a viewable copy of Murchison's book.
Possible antecedent formulations of 'reality check'
It may be that "reality check" initially arose in the sense of a restraint imposed by reality, rather than in the sense of a consultation with reality. However, the only such instances I have been able to find involve juxtapositions of reality as a noun and check as (probably in one case and definitely in others) a verb.
One very early instance of the phrase appears in volume 16 of AERA (a publication of the American Electric Railway Association, an organization of street or transit railways that existed under that name during the years 1910–1932, but has undergone three subsequent name changes and is no known as the American Public Transportation Association). In a sketchily reported policy discussion reproduced in AERA in 1927, the following notations appear [combined snippets]:
Inventory and depreciation.
H. V. FABER, Assistant Treasurer, Jacksonville Traction Company, Jacksonville, Fla. — Where capital sub-accounts are individual jobs it is well to subdivide this information as a basis to be used on future jobs of like nature, and to aid in the future in determining past performance.
W. MONTGOMERY, Treasurer. Madison Railways Company, Madison, Wis. — We have no such accounts.
H. B. CLEVELAND, Secretary and Treasurer, Elmira Water, Light & Railroad Company, Elmira, N. Y. — We do not do this.
A. D. JONES, General Auditor, Interstate Public Service Company, Indianapolis, Ind. — Required by commissions in order to make reality check and compare costs.
E. SMITH, General Manager, The Toledo, Fostoria & Findlay Railway Company, Fostoria, Ohio. — We do not do it but in a large installation, would think it very desirable in order that the expense attached to each individual part of the system might be segregated and clearly set out.
Mr. Jones's wording is not entirely unambiguous. The sense of the relevant phrase could be either "to make [a] reality check and [to] compare costs" or "to make reality check [costs] and [to] compare costs."
A clear-cut instance of "reality check" as a subject and verb where check means "restrain" is from "Truthful Selling is the Basis of Confidence," in Drug Markets, volume 19 (1926) [combined snippets]:
If all automobiles were as fine as Locomobile, they would command the same price.
But does that cold reality check the warm glow of superlatives with which most motor-car advertising is suffused? Any issue of the Saturday Evening Post proves that it does not.
And another, later example of the same type is from Drama,volume 17 (1938), evidently referring to F.L. Lucas's 1935 play, The Lovers of Gudrun [combined snippets]:
The author says, "I wrote the play before going to Iceland. In my experience it is often better to trust first to one's imagination; let reality check it afterwards. In this case the reality proved very like, and almost as exciting. I saw the homesteads of Kiartan and of Gudrun, and the place of Kiartan's killing, and their two graves at Borg and on Holyfell, with the black mountain range of Snaefellsnes lying for ever like a jagged sword between them.
I don't want to pursue this inquiry further, simply because there is no need to suppose that, when the noun phrase "reality check" popped up in psychiatric and psychological use, it meant anything other than "reference made to reality to confirm or refute the plausibility of a fanciful suggestion." "Reality check" in the sense of "reality restraint [on something fantastical]" is not especially far-fetched, but it doesn't seem to be the sense of the phrase in the 1935 citation mentioned earlier.
Though the earliest matches Google Books matches for "reality check" appear 15 to 20 years after the 1935 instance of the phrase the the OED cites (as reported by weissj in a comment above), they have in common with it a psychological/psychiatric setting, and a focus on fantasy versus reality and the "lapsing" of the "reality check" safeguard against error (or delusion). I think "reality check" began as a specialized term in psychology/psychiatry but then escaped the hospital (or asylum) and entered nonspecialist discourse, probably sometime in the middle 1950s or early 1960s.