It's an unusual usage, to say the least. Spoil normally applies to processes that take quite a bit longer than a change of facial expression, even though in this case the sentence explicitly says the change is happening slowly.
It's common, for example, to find reference to things like smoking, long-term drinking, smallpox, wearing spectacles, etc. "spoiling [her] looks".
Having said that, OP's example might turn up as a somewhat contrived metaphorical usage in "flowery" fiction or poetry, but for me at least it wouldn't hit the spot. Except if it were in the unusual context of "over a lifetime" as suggested by @Mitch (some crushingly depressing circumstance lasting for years, perhaps), which I might think of as inventive and striking usage.
LATER: I know this isn't writers.se, but I was never all that happy with "looks" being degraded by a temporary change in expression in the first place. How about A concerned expression slowly cast over his face?