1

I ran across a couple claims that relatively only a few words make up most of the written material in English -- namely, that the most popular 100 words account from a third to a half of all content written in English -- and although it looks plausible enough on the surface, I wanted sources. However, I couldn't find a reputable source anywhere, just the same tidbits repeated all over the place.

The most trustworthy secondary source I could find was this blog post from dictionary.com, but I'd say trustworthy only because you'd assume they'd have some incentive to take associating themselves with this claim seriously.

It seems the problem stems from the relevant research available being from before the internet, or even personal computers, and so it wasn't digitized, only passed down like oral tradition. Apparently there's no recent research on the topic, at least none that I could find.

Can anyone point me to anything more substantial on the topic? Thanks.

4

Oxford Dictionaries has this article that talks about word frequencies takes its statistics from the Oxford English Corpus.

"Instead of talking about words, it’s more useful in this context to talk about lemmas, a lemma being the base form of a word ... Just ten different lemmas (the, be, to, of, and, a, in, that, have, and I) account for a remarkable 25% of all the words used in the Oxford English Corpus ... Similarly, the 100 most common lemmas account for 50% of the corpus, and the 1,000 most common lemmas account for 75%."

The Oxford English Corpus is composed of texts from a wide variety of subjects and mediums, and its purpose, as with any text corpus, is to be able this kind of analysis of the language.

  • 1
    Although one has to be careful, since they analyzed lemmas in the corpus, not words. – Helmar Sep 11 '16 at 16:51
  • That's true, but the lemmas can still give you a general idea of which words are popular. It's probably closer to 150 words for 50%. – 54 69 6D Sep 11 '16 at 17:01
  • 1
    I agree, I just wanted to point out that lemmas are not synonymous to words. Most likely that analysis is the source of what the OP heard. – Helmar Sep 11 '16 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.