milksop is a very similar word, which Merriam-Webster defines as
a weak boy or man
or
an unmanly man
The etymology given is
Middle English, literally, bread soaked in milk. First Known Use: 14th century
A Google search for milksop provides the synonyms:
namby-pamby, coward, weakling, Milquetoast
Presumably, the origin lies in the fact that milk soaked in bread is soft, squishy, and falls apart easily, hence the extension to becoming a pejorative term for someone who is weak. Hence there is no development of milk itself as being a negative term, it is merely sopping wet bread that is used as a comparison to someone who is physically or emotionally weak. Presumably, bread soaked in milk was a dish at that time that this insult developed.