1

I can't find any examples of tokenization/tokenize written with an "s", and MS Word also corrects it to a "z" when set to UK English. Is there a reason this word does not follow the general rule of "sation"/"ise" or is it simply too uncommon to be found online or by auto-correct?

8
  • tokenise is effectively a brand new word (primarily created for / used in the cryptocurrency context; the full OED doesn't even list it yet). The general trend is for BrE spelling to fall into line with AmE, so you could reasonably go with tokenize even if your target readership is British. There's no "reason" beyond that (but note that my initial link into Google Books lists many instances of the "British" spelling). Apr 13 at 12:05
  • Does that mean I can also reasonably use "tokenise"?
    – RnRoger
    Apr 13 at 12:25
  • 2
    @FumbleFingers 'tokenise' isn't all that new; descriptions of the BBC BASIC programming language (developed in Britain for the BBC Micro and released in 1981) mentioned it as part of the process of running programs in the language, and Lexico says "tokenize (British tokenise)". The use is not unknown in the US, for example, a Stanford University article about the Fortran language has "It calls PARSE to tokenise individual records" Arguably the -ise spelling is more common in British English data processing circles. So a CV for lack of research. Apr 13 at 12:39
  • 1
    It's common for (comparatively) new technical terminology to retain the original US spellings when used in the UK. So, for example, we use "tokenization" and "program" in those contexts rather than the British spellings, "tokenisation" and "programme". Apr 13 at 12:50
  • @MichaelHarvey: Argue with the full Oxford English Dictionary, not me. I'm simply reporting the fact that they (still) don't list it. Obviously the word "token" has been around a long time, and it would always be trivial to "ad-hoc verbalise" it. Here's a written instance (with "BrE spelling! :) from 1872 - but at least the writer recognises it mght be a stretch, since he says ...those tokenised if we may use such a term. Apr 13 at 13:30

This site is temporarily in read only mode and not accepting new answers.

Browse other questions tagged .