Merriam-Webster Online's definition explicitly associates button-down with collars:
1 a of a collar : having the ends fastened to the garment with buttons b of a garment : having a button-down collar
2 or buttoned-down : conservatively traditional or conventional; especially : adhering to conventional norms in dress and behavior [button-down businessmen]
and it dates the first known use of the adjective to 1934. However, a Google Books search demonstrates that the usage is at least 18 years older.
From an ad for Wolff's Shirt Shops, in The Yale Literary Magazine (February 1916):
"B 366" "The Student" This Shirt has been voted upon the men of all Leading Colleges "THE SHIRT." The above cut portrays its exactness—it is made of a High Grade Birdseye White Oxford, has a long deep pointed button down Collar with soft French Cuffs,
From Northwest Catalog Co. [Minneapolis, Minnesota], Catalog (1918) [snippet]:
White Summer Shirt 27B4730—Men's shirt made of fine white pongee with watch pockets, turnover button-down collar, imitation French cuffs, cut coat style button over five pearl buttons, cut full and roomy, and is a cool, comfortable summer shirt that can be
From an ad for Buttrick & Frawley, in The Cornell Countryman (January 1921):
Shirts and Neckwear to the King's Taste, and Prices on Both to Yours
Newest Cravats 75¢ Up
White Cheviot Shirts with button-down collar $3 to $4.50
From "Fashion Forcast for Jackets and Coats," in The Boys' Outfitter (May 1921):
Among the novelties is the vestee jacket, as depicted in [illustration] Number 241. This has a button-down collar and outside breast pocket.
From an ad for Yale Co-op in The Yale Alumni Weekly (May 19, 1922):
Do you remember the especially fine white sport shirt you used to buy at the Yale Co-op? We still carry that same make and get orders for these shirts nearly every day from some part of the country. Single or French cuffs—button-down collar. New stock just arrived.
Another early instance spells the adjective without a hyphen—and doesn't refer to collars at all. From an ad in Vanity Fair (1927) [snippet]:
Perfect fit is assured by the small buttondown belt at the back which adjusts to your exact measure. Exclusive with us, these shorts cannot be obtained elsewhere.
A Google Books search also turned up several (but considerably fewer) early examples of "buttoned-down collar," including one especially early (but unverifiable) instance from a 1911 Sears Roebuck catalog.
From "Horse Show Fashions pro-English," in The Clothier and Furnisher (December 1921):
Then, too, the evidence indicated that we have not seen the last of the white shirt with attached, buttoned-down collar. The college enthusiasts are sticking to it longer than usual. An occasional stiff bosom shirt appeared, one of character having light and dark cross stripes of pink.
From "Collars," in The Clothier and Furnisher (April 1922):
Great interest is manifested in the attached collar shirt. These shirts are being produced in Oxford cloth with a range of colors ; the soft, buttoned-down collar giving a certain dress effect.
To sum up, the Google Books search results suggest that "button-down collar" first achieved significant popularity in the northeastern United States among college students in the 1910s and early 1920s, and caught on more generally from there. At least one contemporaneous clothing industry journal (The Clothier and Furnisher) seems to have preferred the spelling buttoned-down, but the majority of haberdashers opted instead for button-down, and that spelling won out. I doubt that the shirt sellers in question based their decision on any grammatical theory.