There's a fairly common expression: "Don't hold it against them", meaning "Don't blame them for what has happened".
But does this exact phrase make sense: "Don't hold me against your decision"?
Context
The other day I didn't do something that I wanted to do, because I knew it would annoy my friend. Talking about it later, she said I should have done it, and "don't hold me against your decision".