I understood the meaning of the phrase to be relatively benign and mostly used facetiously. Can it be viewed as offensive in contemporary conversation?
In the present day, the phrase could be used in a joking way to express disapproval of a newcomer who sets some precedent for change in the social environment.
I would caution, however, that it originated as an expression of resignation and disapproval of racial minorities moving into previously all-white neighborhoods. Key drivers of housing integration in the U.S. include Shelley v. Kraemer, a 1948 case in which the U.S. Supreme Court ruled that racial covenants were unconstitutional; the 1968 Fair Housing Act, which banned discrimination in housing; and court-mandated school desegregation busing which began in the 1970s.
Many in the white majority considered integration undesirable, either because they believed the newcomers would make bad neighbors, or because they believed that white disinclination to live in integrated neighborhoods would mean a decline in property values, and or both. If one minority household moved in, others would soon follow, and the neighborhood, it was said, would go into terminal decline.
This sense of the phrase is far from forgotten. Even if you intend to refer to some other characteristic of a newcomer, it may be interpreted as singling out his or her race, a phenomenon which is the basis for the entire South Park episode “Here Comes the Neighborhood.”
It can definitely be used both ways. I'm not certain of the origin, but the ngram is steady at zero until the early 1960s.
It seems to have originated during and because of the American Civil Rights Movement and desegregation, so I believe it does have racial connotations.