I understood the meaning of the phrase to be relatively benign and mostly used facetiously. Can it be viewed as offensive in contemporary conversation?
In the present day, the phrase could be used in a joking way to express disapproval of a newcomer who sets some precedent for change in the social environment.
I would caution, however, that it originated as an expression of resignation and disapproval of racial minorities moving into previously all-white neighborhoods. Key drivers of housing integration in the U.S. include Shelley v. Kraemer, a 1948 case in which the U.S. Supreme Court ruled that racial covenants were unconstitutional; the 1968 Fair Housing Act, which banned discrimination in housing; and court-mandated school desegregation busing which began in the 1970s.
Many in the white majority considered integration undesirable, either because they believed the newcomers would make bad neighbors, or because they believed that white disinclination to live in integrated neighborhoods would mean a decline in property values, and or both. If one minority household moved in, others would soon follow, and the neighborhood, it was said, would go into terminal decline.
This sense of the phrase is far from forgotten. Even if you intend to refer to some other characteristic of a newcomer, it may be interpreted as singling out his or her race, a phenomenon which is the basis for the entire South Park episode “Here Comes the Neighborhood.”
As choster's answer indicates, "There goes the neighborhood" is intimately associated with racial integration of neighborhoods in the United States during the 1960s, and with the demise of formal and informal deed restrictions and neighborhood covenants that prohibited the sale of homes in certain white neighborhoods to nonwhites. When—through individual defiance of the restriction or covenant or by some other means—a neighborhood ceased to be white only, the result was often white flight and a significant fall in the value of houses in the neighborhood. Hence, "There goes the neighborhood."
But "There goes the neighborhood" is in one way an extraordinary catch-phrase: Although it undoubtedly originated as a genuine expression of concern (or regret or horror) by a homeowner that a newcomer to the speaker's neighborhood was likely to lower the value of houses there, it quickly became an ironic punchline—so quickly, in fact, that the first four confirmable occurrences of the phrase in a Google Books search—all from 1967—are clearly satirical.
From AFL-CIO Committee on Political Education, Memo from COPE (1967) [snippet]:
Sen. Robert Kennedy recently moved his Senate office. It now adjoins offices of Dixiecrat John McClellan and Dixiegop Strom Thurmond. His invasion of this southern preserve prompts Washington line: "Well, there goes the neighborhood."
From syndicated columnist Art Buchwald, Son of the Great Society (1967) [combined snippets]:
When the flying saucers were sighted over Ann Arbor, Mich., a few weeks ago, the first reaction from one of the residents was, "Dammit, there goes the neighborhood."
It has been reliably reported that occupants of all flying saucers are little green men, and this raises a serious problem. Do we want our children to go to school with little green children? What happens to real estate values when the little green people start moving in on the block? Will the green people be responsible for a rise in the crime rate? These questions have to be answered before there is a mass invasion of them and we have a situation that will make Watts [the August 1965 Watts riots in Los Angeles] look like a tea party.
From a review of "Funny, You Don't Look It or (How Can You Say The Whole World Isn't Jewish When Even The Sun's Name Is Sol?)" (released in 1966), in American Record Guide, volume 33, issue 2 (1967) [combined snippets]:
So Paul Revere rouses a lady citizen with his warning that the British are coming. "Oy," she responds, "there goes the neighborhood!" Both Revere and his kindled compatriot, you see, sport burly Jewish accents, as does every other character in this series of would-be hilarious "historical" recreations. The episode is fairly typical of the base-ore humor in store for you in this latest assault on a fun-buff's sensibilities. But you will have to hear the record for yourself (an experiment I am not recommending) to believe its abysmal taste.
From a review of Pete Seeger, Waist Deep in the Big Muddy and Other Love Songs [released August 1, 1967], in Hi Fi/Stereo Review, volume 19 (1967) [combined snippets]:
With a chorus of friends and neighbors from his home town of Beacon, N.Y., he offers a swinging treatment of his own number "Oh, Yes, I'd Climb the Highest Mountain," as well as spirituals and traditional folk tunes. In the course of "Seek and You Shall Find" he pauses for a couple of entertaining anecdotes, including one about an Indian who hears Columbus greeting his tribe in Spanish and mutters, "Well, there goes the neighborhood." He plays a calypso number on his banjo, offers two moving ballads without accompaniment, and winds up the show with a well-intentioned if bathetic number about the civil-rights trio murdered in Mississippi.
All four unique Google Books matches for "There goes the neighborhood" from 1968 are joke punchlines, too. More-recent usage of the catch-phrase seems divisible into three categories: humorous use in a joke setting, as in the 1967–1968 examples; sarcastic use as a view attributed to someone else; and catch-phrase-recognition-based use in display type (chapter titles, section subheads, and the like), where the point is the familiarity of the phrase, which may be only tangentially relevant to what's going on in the text. An instance of the first kind is Andrew Greeley, The Bishop Goes to the University: A Bishop Ryan Novel (2010):
"Half of the Div School students are Catholic and almost half the faculty."
"There goes the neighborhood."
She thought that was very funny.
An example of the second type occurs in Evan Hughes, Literary Brooklyn: The Writers of Brooklyn and the Story of American City Life (2011):
Some see the uniqueness of today's Brooklyn as being under threat or on the wane. Whenever a Starbucks opens, or a chain store, or another real estate broker, you hear the refrain: there goes the neighborhood. A culturally vital neighborhood is a neighborhood where a lot of people want to be, and a place where a lot of people want to be, and a place where a lot of people want to be is a place that's more expensive. Gentrification has its costs, and they are serious.
An example of the third type is this chapter title and subtitle from Anna Soffee, Nerd Girl Rocks Paradise City: A True Story of Faking It in Hair Metal L.A. (2006):
[Chapter] 7. There Goes the Neighborhood
The Smell of Hairspray Gives Way to Teen Spirit
Given its history, "There goes the neighborhood" has always been freighted with racial overtones. But it has also been freighted, from a very early date, with the weight of ironic and facetious usage. Neither form of baggage is benign, in my opinion. If you plan to use the catch-phrase, be aware that it has a peculiar and complicated past, and that it remains loaded today.
Update (March 23, 2021): Early newspaper instances of the phrase
A search of the Elephind newspaper database turns up eight unique instances of the expression "there goes the neighborhood"—one from 1964, two from 1966, one from 1968, and four from 1969. I reproduce the earliest six occurrences below. As in the Google Books search results, even the earliest instances take the form of a joke punchline.
From "TV Scout: Up-to-the-Minute Program Review," in the Madera [California] Tribune (November 26, 1964):
10-11 (Chan. 47) Rowl's attempts to get a dinner invitation for Thanksgiving, and Norm Crosby’s discussion of the Pilgrims, the Indians on the beach ("There goes the neighborhood”) and Queen Isabella gives The Jimmy Dean Show some bright moments. Musical guests are Johnny Cash, Molly Bee and Floyd Cramer.
Norm Crosby was a stand-up comedian whose signature shtick was to deliver a humorous monologue studded with malapropisms ("the pinochle of success").
From Dick West, "The Lighter Side," in the [Palm Springs, California] Desert Sun (September 6, 1966):
I personally do not mind driving faded bilious pink car with bright neurotic pink spots. In fact, I rather like it. But when I drive home in the evenings, the people next door wave their fists and shout, "There goes the neighborhood!"
From Walter Mears, "LBJ Courts N.Y. Voters, Raps GOP: President Cheered, Picketed" in the San Bernardino [California] Sun (October 13, 1966 7, 1966):
"Yet it hasn't been too many years since Italian-Americans felt the raw pain of discrimination in America [President Lyndon] Johnson said. "It wasn't too long ago when, if people talked about slums, they were referring to Italians — and to Irishmen and Jews as well.
"That sad point is made by an old joke of the day," Johnson said. "When two Indians saw the Nina, the Pinta and the Santa Maria sailing toward their shore, one groaned: 'Oh, lord. There goes the neighborhood.'"
Johnson made his remarks at a dinner in Brooklyn, New York, commemorating Columbus Day, but the context of those remarks was political opposition to proposals for a federal Fair Housing Act (which the U.S. government eventually enacted in 1968).
From a pair of cartoons in the San Bernardino [California] Sun (November 6, 1968):
If Nixon Wins: If "There goes the neighborhood!" If Humphrey Wins: "Lyndon! More coffee!"
From "Help a Bluebird Find Happiness," in the Santa Cruz [California] Sentinel (January 29, 1969):
The major reason for the serious trouble in which the bluebirds find themselves seems to be that most of their natural nesting sites have been destroyed or taken over by what bluebird fans call "undesirable foreign birds," namely sparrows and starlings. The bluebirds are unable to compete with these less noble types. As the bluebirds see it, once a starling moves in, there goes the neighborhood.
And from Merriman Smith, "Washington Window," in the [Golden] Colorado Transcript (March 13, 1969):
Backstairs at the White House:
It was said last fall along the quiet beaches of Key Biscayne, Fla. that, upon word President Nixon was establishing a permanent compound on Bay Lane, a long-time islander observed, "Well, there goes the neighborhood. . . .
While this was less than coining a phrase, it did turn out to be somewhat true. For at least one block of Bay Lane, on both sides of the street, the section is now an undisguised security area. Guests must now tell the police and-or Secret Service when they are expecting guests and who they are.
All six of these instances—and the remaining two, which involve, respectively a Martian invasion and the arrival of Columbus in the New World—use "there goes the neighborhood" either as a punchline or as an ironic turn of phrase. Although all of the cited examples are from newspapers in California and Colorado, they were published nationally in syndicated columns or Associated Press wire stories, so they are not specific to those two states.
I agree with everything in choster's answer but the 'originated' bit. There goes the neighbourhood surely comes from the snowclone (http://en.wikipedia.org/wiki/Snowclone) "There goes X", as in
There goes our plan.
There goes the country (down the drain).
There goes all our money.
There goes his dignity.
While there is nothing racist or classist intrinsic in a neighborhood "going" (maybe a flood just washed over it), the connotation of "there goes the neighborhood" is indeed racist and/or classist. Tread with care.