My answer focuses entirely on identifying early instances of the phrase "X wouldn't [or 'would not'] think it to look at Y"—for historical context. The oldest match in an Elephind search comes from "The Low Price of Stock and the High Price of Meat" in the Sydney [New South Wales] Morning Herald (March 23, 1855):
It is noticeable that all the letters that have appeared on this subject in the papers, set forth the same views. They are certainly all on the one side, not even excepting two or three from "E. J. B.," some penny-a-liner, I suppose, employed by the carcase butchers to take up the cudgels in their behalf. But either from the untenableness of his position, or from some natural logical obliquity in his idiosyncrasy, his letters are at least as damaging as any to the cause he meant to defend. His representation of the Sydney carcase butchers as a set of injured and oppressed, but uncomplaining, innocents, is droll. All I can say is—"a body would not think it, to look at them."
From Charles Dickens, "The Second Player," in the [Harrisburg, Pennsylvania] Daily Patriot and Union (March 6, 1861):
"And how did it end?"
"Why, I went to the little beast, titled as he was, and kicked him out at the stage door, I did, sir, though you would not think it to look at me now."
"And the manager?"
"Came and thanked, me. Said he was much obliged to me; he had more annoyance from the complaints of the girls about that fellow than any other cause. He raised mine and my wife's salary that same week."
From Edmund Kirke, "An Old Apple Woman," in the Belmont [Ohio] Chronicle (November 14, 1867), reprinted from Harper's Monthly (November 1867):
"And how old are you?" asked the gentleman, looking at her furrowed face and white hairs, which seemed to say a century.
"Seventy next Christmas. But ye wouldn't think it to look at me. I feel a'most as peart as when I was thirty."
From Mary Braddon, "The Scene-Painter's Wife," in the Mariposa [California] Gazette (February 25, 1870):
"You wouldn't think it to look at her now, sir," said the old clown, as he shook the ashes out of his blackened clay, " but madam was once as handsome a woman as you'd see for many a long day. It was an accident that spoilt her beauty."
From "Some of My Early Adventures," in the [Sydney, New South Wales] Australian Town and Country Journal (December 10, 1870):
"In love," I echoed ; "why, no—I can't say that I have been—that is really overhead and ears, you know."
"Ah, that's it," cried Mr. Chuckle, thumping his broad bosom, "that's it. Over head and ears. That's my state, Mr. Rackett ; though maybe you'd wouldn't think it to look at me."
"I certainly would not," I answered, in all sincerity.
And from Rhoda Boughton, "Good-Bye Sweetheart!" in the St. Cloud [Minnesota] Journal (December 21, 1871):
"Not very often," replies the girl, gravely, looking away beyond him, to where, on the Rance's right bank, Lehon Abbey lifts its roofless walls and gray arches to the sky, "Once, long ago, when I was little, I was very, very ill—I'm not over-strong now, though you would not think it to look at me— and the doctor said I was to have what ever I asked for, for fear of bringing on a fit of coughing if I screamed ; the consequence was that, if ever I wanted any thing, I always threatened to break a blood vessel, and straightway got it."
The idiomatic form "X would not think it to look at Y" thus occurs six times in Elephind search results between March 1855 and December 1871—and three times between February 1870 and December 1871. The fact that all six occurrences appear in the context of quoted dialogue indicates that the form arose and became established in familiar speech before it came into use in formal writing. It is also noteworthy that one of the earliest occurrences of the form appears in a story attributed to Charles Dickens. The rather meager evidence of the six instances cited here suggests that the expression may have originated in England but immigrated to Australia by the 1850s and to the United States by the early 1860s.
