Hoist isn't especially sensible in terms of the metaphor of someone falling victim to their own explosive, over other options like "blown to bits" or maimed or merely killed, hurt or caught.
But it's worth considering the fuller context of the original use:
For 'tis the sport to have the enginer
Hoist with his own petar'; and 't shall go hard
But I will delve one yard below their mines
And blow them at the moon:
Shakespeare has Hamlet follow up this use with a related metaphor of undermining (as in the delve bit here, the mine bit here is mine as in bomb) them and then blowing them "at the moon". Hoist works better in his use of the petard metaphor, because he can then apply the same imagery of being propelled upward to a much greater extent in his next metaphor.
Indeed, if anything here the hoist serves mainly to build up to this "at the moon".
(It may also be because the fart pun is funnier that way as a good fart joke will go down well with some sections of the audience and the bit when you're using vivid metaphors about explosions sending people toward to moon is the place for them. That would explain Shakespeare's decision to drop the d, though it isn't clear to me that people often spoke about farts in French at the time).
Outside of this context, hoist is not as obvious a choice as others, and other figurative uses of petard tend to blow things open or tear them down, as often as upwards, especially since their tactical use was in breaching doors and gates:
Give but fire To this petarde, it shall blow open Madam The iron doores of a judge. — P. Massinger, Unnaturall Combat, 1639
His very name being a Petrard to make all the city-gates fly open. — T. Fuller, Holy State, 1642
Eternal Noise, and Scolding, The Conjugal Petard, that tears Down all Portcullices of Ears. — S. Butler Hudibras: Third Pt., 1678
And for this reason, it would not have been surprising if the phrase had been mutated to "blown up by his own petard" or something similar. But consider the next two citations of the expression's use after Shakespeare's that the OED has:
‘'Tis sport to have the engineer Hoist with his own petard’, as our immortal Shakspeare has it. — Scott Woodstock III. ix., 1858
To see the cruel bibliolater, in Hamlet's words, ‘hoist by his own petard’. — T. De Quincey Protestantism (rev. ed.) in Select. Grave & Gay VIII., 1858
These two both give their source and as such take greater pains to get the quotation right than they might have otherwise. And in bringing the original context to mind they help justify the relatively strange hoist. Indeed they likely do so (given that the phrase was not yet so well known as an expression) precisely because hoist makes so little sense out of context.
And around this time the expression "blown sky-high" came into use, while petards began to drop out of actual military use, making a verb for sending somebody upward more reasonable.
By the time George Eliot came to use it eight years later than De Quincey, it was apparently a phrase known (at least to the sort of people who read Eliot) and used as a unit, and so the unusual verb hoist that originally made sense only in the context of a subsequent metaphor, was now preserved with it. By now many people know petards only as something that hoists.