Arthur Schwartz, Arthur Schwartz's Jewish Home Cooking: Yiddish Recipes Revisited (2008) offers this account of farfel:
Farfel
The word farfel is fun to say, and it has symbolism. It is related to the Yiddish word farfaln, "fallen away" or "over and done with." On Shabbos, when all business and debts are supposed to be settled, and all hard feelings put aside, dishes with farfel take on extra meaning. In High German, on which Yiddish is based, varveln are noodles in the form of pellets or granules, which are what farfel are, too.
Mentions of farfel in this sense show up in Google Books search results at least as early as 1903.
From Isidore Singer, ed., The Jewish Encyclopedia: A Descriptive Record of the History, Religion, Literature, and Customs of the Jewish People from the Earliest Times to the Present Day, volume 4 (1903):
Teigachz, or pudding, of which the kugel is one variety, is usually made from rice, noodles, "farfel" (dough crums), and even mashed potatoes.
From Cornelia Bedford, "March 'Busy Housewife' Menus," in Table Talk (March 1909):
Friday in March ... Supper
Onion Farfel | Cake | Cocoa
From an advertisement that mentions Manischewitz Matzo Farfel in A. T. Philips, Form of Services for the First Two Nights of Passover (1912):
And when you sit down to the Seder table to enjoy the many delightful Passover dishes, you can double your enjoyment when they're made with the these good Manischewitz Matzo Products:
Matzo Meal | Cake Meal | Matzo Farfel | Egg Matzos | Whole Wheat Matzosa | Matzo Cereal
Usage of farfel in the poster's family to refer to lint, fuzz, chads, and similar (mostly fabric-related) effluvia seems highly likely to have originated as an extension of the granular foodstuff meaning of farfel noted here.
A Google Books search for farfel over the period 1954–2019, turns up many matches for the food sense of the word. It also appears (rarely) as a pet name, both figuratively and literally. Andrew Morton, Monica's Story (1999):
As the proud father, Bernie [Lewinsky], himself a doctor, looked on, the nurses who had helped Marcia through her longest day marveled at the beautiful long eyelashes of her seven-and-a-half-pound daughter. Bernie called her "My little Farfel," farfel meaning "noodle."
Farfel was also the name of a puppet dog featured in the stage act of the ventriloquist Jimmy Nelson, according to "Nestlé's Quik," in The American Grocer (1962) [combined snippets]:
One of the most appealing consumer tie-in offers to come along pairs Nestlé's Quik chocolate flavor with "Farfel," a popular TV toy dog "personality," in a fall promotion that is expected to boost sales of Quik. "Farfel," a soft, cuddly, 12-inch-high stuffed dog featured on Jimmy Nelson TV shows, is available to purchasers of Quik who send $ 2.00 plus the code number from any size package of Quik together with their name and address to P.O. Box 316 , Trenton , N. J. The "Farfel" dog, a Nestlé exclusive premium, has a $4.00 retail value.
You can watch a 1956 advertisement for Nestlé's Quik, featuring Nelson, Danny O'Day (a dummy), and Farfel here. Although there doesn't seem to be anything particularly farfelly about Farfel, the Wikipedia page for the puppet reports that "Nelson named [the puppet] Farfel, after the Jewish pasta dish he had seen on the menu of resorts in the Borscht Belt on his tour route."
The Google Books results for 1954–2019 also turn up numerous instances of Farfel as a last name. But there are no reported matches for farfel in the sense of non-food crumbs of detritus (fabric or otherwise) that the poster asks about.