Which would be the correct word order and why? Is there a generic rule that can be used with similar constructs (verb + preposition)? If yes, what is it called?
'Take off' is a phrasal verb meaning here 'remove' (an item of clothing).
Both OP usages are fine.
There can be differences however. Aside from register (putting 'off' after the items to be removed is perhaps a little less formal), the imperative versions tend to be used differently:
Take off your hats (and leave them outside)! Is a simple command. For example, a teacher to a group of school children. Ordinarily there are emphases on 'take' and 'hats'.
Take your hats off! Although this too can be a simple command with emphases as before, this version is usually preferred if the speaker is wanting to clarify and emphasise the direction of the action. In this case the emphasis is placed on 'off' - Take your hats off! (i.e. don't leave them on).