First, the dictionary definitions:
A large payment of money, especially as compensation or a dividend: an insurance payout
Withdrawal count noun
1.1 An act of taking money out of an account.
The word payout is normally used in the sense of someone else paying you for some reason, whether that reason is legal, contractual or ethical, etc.
In your context, the word withdrawal is more suitable - the account holder withdraws funds and may want to cancel a pending withdrawal request.
Here's a PayPal page that uses the phrase "cancel a withdrawal", and here's one regarding the cancellation of a withdrawal that is still pending.