What is another way of saying when you take pleasure in doing something that convention says you are not supposed to do ? (ie the word or expression that I am looking for is very similar to saying for example, I take "subversive pleasure" in talking loudly in the library. I am not thinking of "perverse pleasure", though that is an equally good synonym, its just not the one that I can't think of.)
Consider transgressive pleasure
"Transgressive" definition: going beyond acceptable boundaries of taste, convention, or the law.
Example: consecrating innocence, the modern mythology of childhood ascribes to children a specially rampant natural appetite for all kinds of transgressive pleasures, including above all the sado-masochistic thrills of fear.
Another expression for the behavior you describe is a little rebellion. The term is used to describe a person's behavior that deliberately breaks established social conventions or rules, usually in some harmless way that often has more to do with the person's desire to show defiance or nonconformity than the convention or rule itself.
Here's an example:
The director closed the doors as he left. Gab inched closer to his friend, reluctant to stand beside the coffin. When he did, he noticed they had put Pal in a bright red tie, which he would have hated. Pal never wore ties, even to important functions. It was one of his little rebellions against society, Gab speculated.
[Source: You Will Believe in Love by Homer Starkey]
Sounds to me as though the person you are describing is a contrarian (or at least has contrarian tendencies); that is, one who disagrees with or proceeds against current opinion or established practice.
Now the contrarians who read this posting will undoubtedly say, "But the word contrarian does not carry with it any emotional freight, particularly pleasure," to which I reply, "Yes, but does a contrarian take no pleasure in going against the grain? Isn't one reason why he does so is because of the pleasure it gives him?"
Moreover, take the non-contrarian who may in fact take no pleasure at all in conforming to society's expectations but secretly wishes he could experience the pleasure of going against the grain? In other words, he wishes he could be a contrarian because being one would give him pleasure.
I revel in talking loudly in the library.
n. Boisterous merrymaking or festivity
v . To take great pleasure or delight (usually followed by in)
n. "riotous merry-making"
v. "to feast in a noisy manner"
This only works, of course, if you feel guilty of riotous merry making :)
Not strictly an answer to the question asked but definitely related:
forbidden fruit (phrase)
A thing that is desired all the more because it is not allowed.
‘They had an added cachet of being forbidden fruit: the Stalinists did not approve.’
‘Once something is banned, an even bigger market is created; like most people, the Chinese are attracted to forbidden fruit.’
The emphasis here is on the thing forbidden rather than the illicit pleasure of the rulebreaking, though the two go hand in hand.
The origin is obvious. However, it's worth pointing out that the fruit of the Tree of Knowledge of Good and Evil is not identified as an apple in the text of Genesis and it has been supposed to be, at various times, grapes, a pomegranate, a fig, wheat and even a mushroom.