Take the 2-minute tour ×
English Language & Usage Stack Exchange is a question and answer site for linguists, etymologists, and serious English language enthusiasts. It's 100% free, no registration required.

As you can see from this NGram, the total number of words in the indexed English corpus that were nouns, verbs, adjectives, adverbs, determinants, pronouns, adpositions, numerals, conjunctions, or particles was around 83%.

This could be an infrastructure or programming issue, but assuming it's not, what possible explanation is there that this number is not 100%? Things like interjections are left out, but are 17% of the words in the entirety of Google's indexed literature interjections, or is there a better explanation?

share|improve this question
    
From here, they posted the full list of what tags they use. Did you include all of them in your search? –  simchona May 24 '13 at 1:36
    
Yep. You can go to the link in the original post and double check, if you want. –  Lincoln Bergeson May 24 '13 at 1:39
    
Hmm, interesting. Is it possible NGrams just don't tag everything? –  simchona May 24 '13 at 1:42
1  
This is a database question, Not about English. –  MετάEd May 24 '13 at 1:48
1  
@MετάEd I think the root question is really "what are all the parts of speech", which could still be on-topic –  simchona May 24 '13 at 1:51

1 Answer 1

up vote 4 down vote accepted

Algorithmically speaking, identifying a word or phrase as a particular part of speech is not possible in all contexts. A word functions as one part of speech in a context and another in a different place, even without any morphological changes.

To make matters more difficult, fiction thrives on using words creatively, leaving the reader to fend for himself as to how a word is to be interpreted or in how many different ways.

The statistics probably reflect that up to about 17 per cent of the words could not be categorically determined to belong to one or the other part of speech -- not that they do not belong to any of the known parts of speech.

Between circa 1523 - 1650 AD, the number of such unaccounted-for words scales vertically from (14.2) to (83.1) where it settles down and remains to date (2007: 83.9). Probably because the English language, especially in fiction writing, was not "normalized" until very recently.

Hm... as you can see this is a mere hypothesis and I could be wrong, even.

share|improve this answer
    
I still have my doubts -- why should the figure hover around 80% all the time? What's special about that magic number? –  Kris May 24 '13 at 5:54
    
I was just about to comment thus on the question: "It's possible (probable?) that their automated heuristics can't positively identify all words as a particular POS, so they are left as unidentified. (For example, what is unidentified in that sentence?)" –  Andrew Leach May 24 '13 at 5:55
    
@AndrewLeach 'Automated heuristics' are built from human knowledge and theories, strengthened by statistics. Even the human reader cannot always tell for sure what a word is: 'For example, what is unidentified in that sentence?' :) –  Kris May 24 '13 at 5:59
    
Exactly so: I was agreeing (and have upvoted accordingly). –  Andrew Leach May 24 '13 at 6:01
    
It's an -ed word. –  Edwin Ashworth May 24 '13 at 8:30

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.