A women's chest is also called her breast but is typically only used with idioms:
She beat her breast.
I rest my head upon her breast.
Obviously, there is a bit of awkwardness regarding these sayings due to the specific meaning of individual breasts. But this actually isn't restricted to women either. A man also has two breasts that would correspond to each of his two nipples.
In short, the only difference between these words with regards to gender is that most people don't refer to male breasts — either as a whole or individually. This isn't a restriction of the words; people just don't make a big deal out of male breasts.
That being said, when they are being specifically referenced they are more commonly called pects or pectorals (which, again, is not a gender specific term) or the derogatory term man boobs which describes someone with sagging, overweight breasts.