Since buttons aren't particularly cute (IMO), where did this common phrase come from? I know it's old; I've seen it in 19th century literature.
While I can't find any scholarly answers, most answers I'm finding say that 'button' refers to something pretty or attractive in a dainty way. After all, you're using the word 'cute' so you wouldn't be using it to describe a large, muscular man. This phrase would be best suited for a small child or flower.
You can read more here.
Years ago I read in an old volume (early 20th century) the expression "cute as a button quail". It was a children's book originating in the United Kingdom, an anthology of prose, written in a more Victorian dialect. I can't remember the name for the life of me, sorry. I don't know if this usage is related to the "cute as a button" etymologically, but button quails are quite small, and the association with smallness seems to be common. Maybe this will jog someone's memory...
I can think of three explanations, not necessarily mutually exclusive:
Obviously if used for decoration it is going to be pretty and small-ish.
as for cute as a bug in a rug... wrong. SNUG as a bug in a rug is correct. Who would think a bed-bug or similar would be cute... and it doesn't rhyme anyway.
Although I agree with the author of the Phrase Finder remarks (cited in skarz's answer above) that two other phrases—"cute as a bug's ear" and "bright as a button"—are highly relevant to an inquiry into the origin and sense of "cute as a button," I disagree with several of his or her conclusions.
When did the three phrases originate?
First, the dates given for all three phrases are off by a significant amount: Phrase Finder dates "cute as a bug's ear" to 1930 and "cute as a button" to 1946; it doesn't give a date for "bright as a button," but describes that phrase as "the British version" of "cute as a button," which many people (I imagine) would interpret as indicating that it is roughly the same age as "cute as a button."
A Google Books search, however, finds instances of "bright as a button" going back to 1805. Here are the earliest three, in chronological order. From "Yankee Phrases," a poem consisting largely of similes drawn from the northeastern United States, in The Spirit of the Public Journals; or, Beauties of the American Newspapers, for 1805 (1806):
From William Stone, "The Mysterious Bridal," in Tales and Sketches: Such as They Are, volume 2 (1834):
And from Nathaniel Hawthorne, "The Story Teller" (later republished as "Mr. Higginbotham's Catatrophe") in The New-England Magazine (December 1834):
Notably all three of these instances are from the United States, as are additional matches from 1835, 1841, 1857, 1863, 1869, and later. The earliest British English instance of "bright as a button" is from Rudyard Kipling, "Private Learoyd's Story" (1888), in Plain Tales from the Hills, 1886-1887: Soldiers Three and Other Stories (1899).
So whether "bright as a button" was ever the British version of "cute as a button" it appears to have begun as an American saying, no later than the very early 1800s.
The phrase "cute as a bug's ear" has a first occurrence date in the Google Books database of 1913. From "Long Island Sound Championship," in Forest and Stream (January 18, 1913):
The phrase "cute as a button" debuts in Google Books in 1931. From a letter of Christmas shopping hints dated December 1, 1931, in Indiana Telephone News, volume 21 (1931) [combined snippets]:
To recap, the earliest Google Books matches for the phrases are "bright as a button" 1805, "cute as a bug's ear" 1913, and "cute as a button" 1931. "Bright as a button" is more than a century older than either of the other two, and "cute as a bug's ear" is about 18 years older than "cute as a button."
What do the the phrases mean?
Christine Ammer, The Facts on File Dictionary of Clichés, second edition (2006) has this entry for "cure as a button":
So (according to Ammer) "cute as a button" means "daintily attractive." Is that what "bright as a button" means in British English? I see no reason to think so. John Ayto, The Oxford Dictionary of English Idioms, third edition (2009) has this entry for "bright as a button":
The same double meaning is discussed in a memorable exchange between the Scarecrow and a lost (and not very well-informed) little boy named Button-Bright in L. Frank Baum, The Road to Oz (1904):
Where did the phrase 'cute as a button' come from?
"Cute as a button" originated in the United States, as did "bright as a button" and "cute as a bug's ear." Since both of those other phrases were well established in U.S. English at the time that "cute as a button" emerged, I think it isn't at all far-fetched to suggest that "cute as a button" may be the product of a merging of the daintily attractive "cute" from the "bug's ear" simile and the small but visible "button" from the "bright as" simile.
The Phrase Finder commentary that serves as the basis of skarz's answer above was written in 2005—before Google Books Ngram searches were available, I believe—and in any event seemingly without reference to any other searchable database of texts that might have turned up more-accurate details about how old the three phrases are and where "bright as a button" originated. It's still an interesting answer, but I think its conclusions are flawed and out-of-date.