Take the 2-minute tour ×
English Language & Usage Stack Exchange is a question and answer site for linguists, etymologists, and serious English language enthusiasts. It's 100% free, no registration required.

Let's consider a partition of the words in the english language according to respective use frequency.

Looking at the frequency graph it should be easy to find classes of words with approximately the same frequency.

Now I'm interested in the size of these classes.

I know that the size depends on the relative cut-off points, but I'm more interested in a general classification, i.e. are there more common words, or more uncommon words?

share|improve this question
    
This is a fascinating question, but is more relevant to a non-English specific text processing site or maybe even linguistics.SE. –  Mitch Jul 16 '12 at 14:38
1  
Your question is unclear. When you say 'classes of words' do you mean classifying together words whose frequency is within a certain range and you'll a priori specify thresholds of frequency? (e.g. 10-5%, 5-3%, 3-2%, and less than 2%?). Experimentally, most human languages follow a Zipf law. –  Mitch Jul 16 '12 at 14:41
    
I can certainly say there are such words, remembering matching both words and letters frequency in substitution ciphers. Now, beyond the most common "the" I can't give you any specifics, but generally, frequency analysis is big both in cryptography and in prediction (like T9 dictionary for mobile phones, where most frequently used words are suggested first.) –  SF. Jul 16 '12 at 14:45
1  
This is actually a Computational Linguistics question, but I fear that StackOverflow would toss it off to Linguistics.SE — and vice versa. There are textbooks that devote many chapters to this sort of thing. You might should look into tf*idf, and see where that takes you. –  tchrist Jul 16 '12 at 14:45

1 Answer 1

up vote 6 down vote accepted

There is a well-known formula that appears to describe the frequency distribution of English words reasonably well:

Zipf's law states that given some corpus of natural language utterances, the frequency of any word is inversely proportional to its rank in the frequency table. Thus the most frequent word will occur approximately twice as often as the second most frequent word, three times as often as the third most frequent word, etc. For example, in the Brown Corpus, the word "the" is the most frequently occurring word, and by itself accounts for nearly 7% of all word occurrences (69,971 out of slightly over 1 million). True to Zipf's Law, the second-place word "of" accounts for slightly over 3.5% of words (36,411 occurrences), followed by "and" (28,852).

Either from actual measurements of word frequency (see, for example, Wiktionary: Frequency lists) or from Zipf's law, one can deduce that most words are infrequently used; there are tens of thousands of words that taken all together are used less frequently than any of the or of or and. Indeed, the most-frequently-used 500 words account for more than half of all usage, leaving hundreds of thousands of words used quite infrequently indeed.

Zipf's law predicts that relative frequencies decrease like 1, 1/2, 1/3, 1/4 ... If there are a quarter-million words in the English language and their frequencies follow Zipf's law, then their relative frequencies add up to 1+1/2+1/3+1/4+...+1/250000, which is about 13.01, ie ln(250000)+gamma, where gamma is the Euler-Mascheroni constant, about 0.5772156649. Given n words in toto, if you want to find out the number of words in each of k groups with total usage the same in each of those groups, solve ln(mi)+gamma = i*(ln(n)+gamma)/k for i from 1 to k-1. That is, mi = exp(i*(ln(n)+gamma)/k -gamma). For example:

• With n=250000 and k=2: m1= exp(13.00643/2-gamma) = exp(5.926) = 374.65, implying that the first 375 words of 250000 will receive about 50% of all usage, if Zipf's law applies.
• With n=250000 and k=5: m1= exp(2.024) = 7.569, m2= 102.04, m3=1375.6, m4= 18544.5, implying that the first 8 words will receive about 20% of all usage, the first 102 about 40%, the first 1376 about 60%, and the first 18545 about 80%, if Zipf's law applies.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.