I think that the source of the idea of *vanilla* as bland and neutral is the expression "plain vanilla"—signifying vanilla as the primary flavoring (of a pudding, ice cream, cake batter, or other dessert). One of the first recorded instances where "plain vanilla' is used in this sense is in Nell Nelson, "[Bun's Garden Party](https://chroniclingamerica.loc.gov/lccn/sn83030193/1890-07-03/ed-2/seq-2/#words=%22plain+vanilla%22)," in the *[New York] Evening World* (July 3, 1890): >Use pitted cherries, red currants and bits of apple, apricot, pineapple, apple and melon. Candied fruits are very nice and half a pound will fairly gem a three gallon can of cream and convert **plain vanilla** into Neapolitan, EVENING WORLD, Sick Baby cream or any other title you care to bestow. For some reason "Sick Baby cream" never caught on—but "plain vanilla" as an undistinguished dessert base did. Another early instance of "plain vanilla" recommends using the psychology of food coloring to jazz up simple desserts. From "[Red Sugar and Some Ways to Use It](https://trove.nla.gov.au/newspaper/article/71391607?searchTerm=%22plain+vanilla%22)," in the *[Sydney, New South Wales] Australian Town and Country Journal* (November 10, 1900): >Ice-cream also is much more attractive if varied occasionally in appearance, and the first time it is served tinted to a dainty shade of pink there will be a general inquiry as to what gives the peculiarly delicious flavor. Imagination will naturally suspect the presence of strawberries; raspberries, or other fruit. In reality, a small portion of red sugar was the only addition to the **plain vanilla** cream, the flavor, however, seeming exceptionally fine. So white is a problem and vanilla is a problem—and the way forward is to introduce additional colors and flavors. ---------------------- **[[UPDATE (June 25, 2018):** Ace researcher RaceYouAnytime has found an even earlier instance of "plain vanilla" (actually, "vanilla plain"), with the implication of simplicity and purity made explicit. From "[Woman's Pet Tipple](https://chroniclingamerica.loc.gov/lccn/sn86064205/1887-08-31/ed-1/seq-8/#words=%22just+vanilla,+plain%22)," in the *Fort Worth [Texas] Daily Gazette* (August 31, 1887): >You see, it's this way: The school girls all want strawberry and vanilla mixed, and the dark ones coffee or chocolate, and the blondes they take pineapple or lemon, and the old ladies call for sars'p'rilla, 'cause it’s cooling to the blood, and the girls who come in with fellows want 'just **vanilla plain**'—kind of innocent and simple, and the young widows always ask for Vichy, with 'a touch of lemon.' The source of these insights is a waitress who works at a soda fountain.**]]** -------------- As the website quoted in lbf's answer observes, one early figurative use of "plain vanilla" is in "[Willkie Evolves a Plain Vanilla Foreign Policy for Republicans](https://books.google.com/books?id=UUEEAAAAMBAJ&pg=PA36&dq=%22plain+vanilla%22&hl=en&sa=X&ved=0ahUKEwiJmqPPqe7bAhUnhlQKHZePD_oQ6AEIRzAG#v=onepage&q=%22plain%20vanilla%22&f=false)," in *Life* magazine (October 19, 1942), which, aside from its title, offers this figurative passage: >Whether rightly or wrongly, many a Republican Congressman voted a straight isolationist ticket before Pearl Harbor chiefly because he could not see a realistic solution to the international problem in any statement from the New Deal. The able Congressman John Vorys of Ohio, for instance, while apparently a strict Republican isolationist, has on a number of occasions expressed his open-mindedness toward international solutions, providing only they are real enough to work. And Mr. Willkie's good and generous friend, Congressman Charles Halleck of Indiana, might be another case in point. Certain it is that patriotic Republicans, such as both of these men are, would welcome a **plain vanilla** Republican solution to the mighty international problem which they must squarely face. Another early figurative instance again involves political parties, but this time Democrats are "plain vanilla." From "[John White Issues Blast at Shivers](https://texashistory.unt.edu/ark:/67531/metapth284023/m1/1/zoom/?q=%22plain%20vanilla%22&resolution=2&lat=3455.01363710025&lon=3205.581479236362)," in the *Sweetwater [Texas] Reporter* (January 8, 1954): >The only state official who didn't run on both the Democratic and Republican tickets last fall said Thursday night "cleanly - drawn" political battlelines would separate "a lot of good, **plain, vanilla** Democrats" from "a few raspberry Republicans." >Agricultural Commissioner John C. White, 29. devoted most of his speech to Tarrant county Democrats to criticism of the Texas' system of cross-filing and political hybrids. Then comes "plain vanilla" in the sense of "garden-variety" or "dime-a-dozen" athlete. From "[Hoot Gibson Proves Grit, Determination Pays Off](https://texashistory.unt.edu/ark:/67531/metapth95790/m1/4/zoom/?q=%22plain%20vanilla%22&resolution=4.010410732164926&lat=3717.425415738345&lon=1813.135339518144)," in the *[Abilene, Texas] Optimist* (September 17, 1954): >When February ro[l]led around and spring drills, Herb checked out his uniform and went through some more hard bumps. >After Spring drills were over he was more or less still classified as a **plain vanilla** scrub. And again in the context of sports, from "[After-Game-Party Held; What a Game And What a Party](https://texashistory.unt.edu/ark:/67531/metapth98433/m1/1/zoom/?q=%22plain%20vanilla%22&resolution=4&lat=3412.5&lon=2333.5)," in the *[Abilene, Texas] Brand* (February 28, 1958): >The basketball victory (81–62) played second fiddle to the school spirit displayed by both Midwestern and the Cowboy cheering section. >What would normally have been a **plain vanilla** game burst into a wide open affair mostly because of the cheering fans. In short order, "plain vanilla" became a descriptive term for all sorts of inedible things—to the [human beings who serve on censoring boards](https://texashistory.unt.edu/ark:/67531/metapth243427/m1/1/zoom/?q=%22plain%20vanilla%22&resolution=4&lat=3803&lon=2759.5) (in the *Crosbyton [Texas] Review*, August 7, 1958); to [theatrical entertainment](https://texashistory.unt.edu/ark:/67531/metapth95956/m1/5/zoom/?q=%22plain%20vanilla%22&resolution=4&lat=2329.5&lon=2199) (in the *[Abilene, Texas] Optimist*, November 20, 1959); to [low-cost automobiles](https://texashistory.unt.edu/ark:/67531/metapth254636/m1/15/zoom/?q=%22plain%20vanilla%22&resolution=6&lat=6911&lon=2342) (in the *[Fort Hood, Texas] Armored Sentinel*, May 27, 1960); to [electric ranges](https://texashistory.unt.edu/ark:/67531/metapth281828/m1/8/zoom/?q=%22plain%20vanilla%22&resolution=4&lat=3956.5&lon=2934) (in the *Crosbyton [Texas] Review*, February 8, 1962); to [a shirt color](https://texashistory.unt.edu/ark:/67531/metapth314203/m1/3/zoom/?q=%22plain%20vanilla%22&resolution=4&lat=3547&lon=2858) (in the *[Denton, Texas] Campus Chat*, March 23, 1962); to [teachers](https://texashistory.unt.edu/ark:/67531/metapth46422/m1/12/zoom/?q=%22plain%20vanilla%22&resolution=4&lat=3691.0941177720233&lon=2430.4058822279767) (in the *Tulia [Texas] Herald*, April 12, 1962); to [everyday people](https://texashistory.unt.edu/ark:/67531/metapth314231/m1/2/zoom/?q=%22plain%20vanilla%22&resolution=4&lat=4534.032259914714&lon=1981.6774199715715) (in the *[Denton, Texas] Campus Chat*, October 10, 1962); to [football formations](https://texashistory.unt.edu/ark:/67531/metapth281863/m1/1/zoom/?q=%22plain%20vanilla%22&resolution=4&lat=3995&lon=2995) (in the *Crosbyton [Texas] Review*, October 11, 1962); to (finally) [sex education](https://thecatholicnewsarchive.org/crra?a=d&d=cst19650618-01.1.7&txq=%22plain+vanilla%22), although not sex itself (in the *[Philadelphia, Pennsylvania] Catholic Standard and Times* (June 18, 1965). The predominance of Texas periodicals in the preceding paragraph is not a sign that Texas was a hothouse of "plain vanilla" idiomatic experimentation; it just underscores how limited other states' accessible post-1920s newspaper databases are. The Portal to Texas History archive is restricted to small-town newspapers and college newspapers—no big metropolitan dailies with their many copyrighted and syndicated stories from wire services and prominent national newspapers—but it still manages to host a respectable amount of relatively recent searchable content. In any event, it seems unlikely to me that the writers of these various instances from the period 1942–1965 were taking their cue on "plain vanilla" (knowingly or not) from the world of sexual preferences. As I noted earlier, the original idea of "plain vanilla" was not that it was unappealing in itself, but that it could serve as a base for a multitude of interesting variations through the addition of other ingredients. Inevitably, however, the relative simplicity of "plain vanilla" invited critical dissatisfaction based on the predictability and unimaginativeness of "plain vanilla." This pejorative element certainly lingers as an overtone of describing something or someone today as "vanilla."