The OED entry (updated March 2003) for pear-shaped, adj. (paywalled), defines sense 3 as
colloquial (chiefly British, orig. R.A.F. slang). to go (also turn) pear-shaped: to go (badly) wrong, to go awry.
OED's earliest attestation is from 1983, in the context of the British response to the Argentine invasion of the Falkland Islands:
J. Ethell & A. Price Air War South Atlantic 158 There were two bangs very close together. The whole aircraft shook and things went 'pear-shaped' very quickly after that. The controls ceased to work, the nose started to go down.
I was unable to confirm any earlier use in the RAF than that attested in 1983.
A use in 1968 (The Post-Crescent, Appleton, Wisconsin, 28 Apr, p. 79; paywalled) attests the earlier appearance of the sense:
And the way the world (a big weather balloon that leaked) went pear-shaped during a performance of "Love Makes the World Go 'Round."
Slang use of 'to go (also turn) pear-shaped' with the meaning "to go wrong, to go awry" in the RAF during World War II may derive from reports such as this 1938 (paywalled) description of the sinking of the US Gunboat Panay (Star Tribune, Minneapolis, Minnesota, 01 Jan, p. 12):
Then, from under the plane, a black dot appears...As it is darting toward you it becomes larger...Then there is a shape to it...A second later...it becomes a pear-shaped plummet, stream-lined for speed, trailing off in a fish-tail.
Hell from heaven, this plummet is a roaring bomb.
Indeed, bombs and mines were frequently described in UK newspapers as 'pear-shaped' during the war.
The mine, a large black pear-shaped object....
Newcastle Journal, 02 March 1940, p. 12 (paywalled).
It was a pear-shaped mine which when dropped from the air had a parachute attached to the "stalk" to deaden the shock of falling.
Belfast Telegraph, 02 March 1940, p. 7 (paywalled).
Examples of what is believed to be a new type of aeroplane bomb of enormous potency are being washed up...The bomb consists of a pear-shaped object three feet long and fifteen inches in diameter.
Liverpool Daily Post, 30 March 1940, p. 7 (paywalled).
In contrast to OED's derivation of 'to go pear-shaped' from RAF slang, Robert W. Holder, in How Not to Say What You Mean: a Dictionary of Euphemisms (Third Edition, 2002; originally published in 1987, revised for later editions) says this:
pear-shaped unsuccessful
Probably from the form of an analyst's graph, the use having started as jargon in financial circles. As with the fruit, the weight is at the lower end...
Holder provides an attestation of the variant "went pear-shaped" from the Daily Telegraph of 20 June 1997.
A common trauma unrelated to the shapes of bombs may contribute to the comparative frequency of 'to go pear-shaped', and so to other constructions on the same model, in contemporary speech. The trauma, male menopause, is described in a 1942 Arizona Republic article:
One common symptom in men is the increase in weight, particularly in lower abdomen and often a loss of fat and muscle also about the chest and shoulders. Instead of the wide shoulders and narrow or medium hips, he becomes pear shaped in appearance.
Women, it appears, suffer from a similar problem. None other than Emily Post, in 1932 (a syndicated column; paywalled), remarks, with phrasing identical to a variant of the later, more abstract figurative expression, that
Precisely as one should take setting-up exercises every day to keep one's figure from going pear-shaped in the hips....
Newspapers reran Post's 1932 advice in 1941, and again in 1949. The same phrasing characterizes other quasi-literal uses in the 1960s:
...[a black-and-white television] has developed a variety of tube troubles in old age. The picture often tends to squish in at the top, broaden at the middle, and then becomes narrow and curvy toward the bottom....
Well, a little aberration can be a devastating thing. What our TV did to thos 36-24-36 figures was bad enough to gladden the hearts of all overweight, over-thirty types.
The pageant wasn't boring, Miss Alexander, it was (if you'll pardon the expression) a riot. Imagine those dolls coming on looking so peachy and then suddenly going pear-shaped!
Garden City Telegram (Garden City, Kansas) 16 Oct 1968, p. 4 (paywalled).
Diamonds may be a girl's best friend after her figure goes pear-shaped, but they can be next to worthless too....
The Californian (Salinas, California) 10 May 1969, p. 1 (paywalled).
Also worth noting is the occasional mention of the 'pear-shaped problem'. The mentions are ill-defined, but show up earlier, bracket Post's quasi-literal uses, and may be the noun progenitor or a sibling of the verb phrase. Here are some examples:
An average of one child a day is killed by automobiles in Greater New York, showing that speed regulations and rules of the road still afford a large, pear-shaped problem for those who must solve it.
The Atchison Daily Globe (Atchison, Kansas)
30 Jan 1913, p. 4 (paywalled).
Now Iverson runs a posture clinic here. He concentrates on children because he believes the answer to the nation's pear-shaped problem lies in correcting youngsters early.
Press and Sun-Bulletin (Binghamton, New York) 18 Aug 1950, p. 13 (paywalled).
"When one monkeys with Uncle Sam's security procedures, he's bound to have a large, pear-shaped problem on his hands," he continued.
The Hays Daily News (Hays, Kansas) 06 Jan 1972, p. 4 (paywalled).
All told, then, the absence of early direct primary source evidence for a British, RAF origin of 'to go pear-shaped', and the comparative abundance of evidence, in the form of identical phrasing, suggests an origin in the earlier quasi-literal expression of the sorry effects of aging on the human physique.