Speaking from an American-culture perspective, instead of a purely grammatical perspective, when someone asks you where you work, they are trying to get to know you. It's an invitation to open up and tell them about yourself. If you say something short like "I work at a hospital" and leave it at that, you are telling them that you don't really want to talk to them.
Of course this all depends on context. It's possible that they just want directions to your place of business.
edit: To answer the second part of your question:
You can say "I work at a hospital" and "I work in a hospital". Both are equally correct.
You can say "I work at McDonalds", but it's incorrect to say "I work in McDonalds".
You can say "I work in New York City", but it's incorrect to say "I work at New York City".
You can say "I work in medicine" but you can't say "I work at medicine".