Most likely, the word you're looking for is an adjective modifying smile or grin, as I can't think of a single word for smile that means what you want.
One option is rueful smile. Rueful literally means feeling pain or regret. But Wiktionary and the Oxford English Dictionary both note that the connotation is sometimes of wry humor, or humor within the regret:
Causing, feeling, or expressing regret or sorrow, especially in a wry or humorous way. (Wiktionary)
- Expressive or suggestive of penitence, remorse, sorrow, or regret. Now also: expressing sorrow or regret in a humorous way; wryly apologetic. (Oxford English Dictionary)
2009 It is embarrassing to admit it, but most people we talk to respond with rueful smiles of recognition. J. Olds & R. S. Schwartz, Lonely Amer. vii.124
Another example is from The Guardian, where Enrico Letta, then prime minister of Italy, gets vital support against a vote of no confidence from Berlusconi. He grins. But, behind his rueful grin, we may see that his situation is precarious or inconvenient:
There's applause in the Senate as Berlusconi says he will support Enrico Letta -- although I think I saw Letta pull a rather rueful grin.
And if this answer doesn't seem to quite fit what you want, it provides a formula for getting to other options: wry smile, bittersweet smile, rueful grin, and so on.