Relish, gratification, GLEE
Chagrin (Merriam Webster), could be used to emphasize the vexatious nature of a feeling, as in the example's opposite, "...to my chagrin, he was able to do it easily".
A list of antonyms of chagrin on power thesaurus includes relish:
relish [noun]...
3 2 enjoyment of or delight in something that satisfies one's tastes, inclinations, or desires (MW)
So you could say:
But to my relish, he couldn't do it either!
Gratification (C.E.D.) also combines meanings of pleasure and personal satisfaction. With slight rewording for emphasis you could also say:
Much to my gratification, he couldn't do it either.
Another word which may be closer to what you are looking for could be:
Glee
1. [mass noun] Great delight, especially from one's own good fortune or another's misfortune. ( via Lexico)
As a native speaker I would say that glee can also denote joy or joyful anticipation of something that has a component of self interest or triumph for the person being gleeful. E.G. "The merchant approached the rich customer who had entered the shop, rubbing his hands with glee".
The ambiguity of how much of glee, if any, is arising from self interest is often left open for a reader or listener to imagine - and could be meant to be deliberately wry.
Hence
No matter how I tried, I couldn't complete the seemingly simple task. So feeling like an idiot I gave up and asked the expert. But to my glee, he couldn't do it either!