I'm not sure whether it will be construed as off-topic either, 978, but it's something I have often struggled with too, and I'm pretty sure we're not in a small minority.
I'd say that the two things to avoid like the plague are:
(1) Sounding sarcastic. "Oh, well done!" is used antiphrastically so often that one has to snap to it to recognise the straight sense. It is often better with close friends to use irony deliberately: "100% ? Oh well, better luck next time!" But this needs real care and judgement - I guess we've all got this embarrassingly wrong on occasion.
(2) Sounding insincere, perfunctory, or (worse) patronising. This is perhaps harder to get past. Again, humour - light irony - can be a help, but has to be watched with extreme care. A technique is to be mildly self-deprecating: That's a great piece, but I could have finished it in twice the time." Only the 'proverbial nice person' or those in an even higher category seem able to get away with a straight "Oh, well done, Miss Granger!" / "Well done, good and faithful servant!"
But don't knock single adjectives / exclamations. And the more 'rarefied', probably the more acceptable - "Good" sounds rather cheesy, "Nice" better if a bit affected, and 'Wow' pretty genuine. "Not half bad" usually has people examining their own linguistic ability rather than the speaker's meaning.