It is quite common to describe a man as slim. In fact, it can even be a cool-sounding nickname: Slim Pickens, Iceberg Slim, and so on.
Slimness can in fact be attributed to a man in a complimentary way, but it depends very much on the context (whereas it is complimentary by default to describe a woman as slim).
For a man, slim is good when compared to fat, bad when compared to athletic or manly (for example).
An athletic physique, implies good muscle tone and posture as well as slimness, so it's not a synonym.
Some good alternatives, that are considered positive things for a man to be, in general, are: lean; wiry; and svelte.
Lean: (of a person or animal) thin, especially healthily so; having no superfluous fat.
Wiry:(of a person) lean, tough, and sinewy.
Svelte: (of a person) slender and elegant.
Be aware though that wiry includes connotations of strength and toughness, as well as just slimness - and that svelte is perhaps more associated with women than men (but it is a compliment to anyone).
-- Of course, all of this depends on cultural beliefs about sex, gender, and ideal body shapes, etc. These beliefs vary from place to place and between generations, so there can be no fully definitive answer to your question. --