Both "yum" ("delicious!") and "yum-yum" (a delicious thing) are typically ascribed to people imitating baby-talk. "Yummy" is simply "yum" in adjective form.
Most early human vocalizations are simple and repeated, e.g. "Mama", "Papa", "Baba", etc. Babies are thought to express pleasure with their food by saying, "Yum!" or "Yum-yum!" This association is often used to indicate pure/primal enjoyment, even by a well-experienced speaker of English.
Given its association with babies and young children, however, most people would not be quick to use it duringin a formal setting. That being said, there are certain foods, such as candy or desserts, which are also associated with children. When eating those, many would allow themselves to revert to such childhood expressions of enjoyment. Some might not, for fear of seeming less mature, elegant, educated, eloquent, composed, or, in the case of men, masculine, but the level of concern over this seems to be in decline.
Short answer: In informal situations, it is not unusual at all. In formal situations, some discretion is called for, but is not always necessary.
Edit: I should emphasize that I am speaking of "yummy" only in terms of food enjoyment, and not as a more open-ended term for things people find very appealing. That usage is extremely unusual here in the USA and would sound to most like a British-ism.