“Abstract mathematics” as used here refers to how mathematics was being taught when Donald Knuth first taught a “concrete mathematics” course. In the paragraph you quote from, we find (with emphasis added)
... concrete classical results were rapidly being swept out of the modern mathematical curriculum by a new wave of abstract ideas popularly call the "New Math." Abstract mathematics is a wonderful subject, and there's noting wrong with it: It's beautiful, general, and useful. But its adherents had become deluded that the rest of mathematics was inferior and no longer worthy of attention. The goal of generalization had become so fashionable that a generation of mathematicians had become unable to relish beauty in the particular, to enjoy the challenge of solving quantitative problems, or to appreciate the value of technique. Abstract mathematics was becoming inbred and losing touch with reality; mathematical education needed a concrete counterweight in order to restore a healthy balance.
That is, the authors assert that there was a self-reinforcing mathematical culture that glorified abstract mathematics while denigrating concrete classical results. They assert that “a generation of mathematicians” had been taught to snub “the rest of mathematics” except for abstract mathematics, with the implication that those mathematicians would go on to teach the same thing to their students in turn.
The applicable sense of inbred is “(often pejorative) having an ancestry characterized by inbreeding”, where inbreeding here refers figuratively to a closing in of horizons because of only considering too-closely-related ideas. This closing in of ideas is parallel to the consequences of genetic inbreeding, such as decreased fitness of a population due to failure to avoid “expression of deleterious recessive alleles”.