One is OK, the rest are not.
- The never spoken rules of using escalators.
This is OK, although we could replace 'never spoken' with 'unspoken' to reduce the waffle factor.
- The never spoken rules of escalators using.
This is not OK, because both 'use' and 'usage' are more likely candidates. Secondly, the use of a plural here has no meaning since 'escalators' is effectively in an adjective position, and the plural form here has to meet certain criteria to be acceptable.
- The never spoken rules of escalators usage.
This is not OK. First we have the plural 'escalators' issue, plus the choice of 'usage' rather than 'use'. Use of the 'usage' is common in describing general uses rather than actual events. This means it rather suggests why the architect chose to install escalators instead of stairs.
We could use:
The never spoken rules of escalator use.
- The never spoken rules of usage escalators.
This is not OK, as the pairing 'usage escalators' is unlikely in comparison to 'using escalators', and suggests more the installing of escalators over the installing of stairs.