0

I am building an automated system to seek out the proper nouns from a piece of text. I have some algorithms available to me that can correctly determine the POS tag of a word in some text. The problem is that for proper nouns, the algorithms rely heavily on casing.Basically, they look at words starting with uppercase and assume that the word is a proper noun.

In my case, the text can have improper casing so its quite hard for me to determine proper nouns. I was wondering if it would be better to look at words surrounding the proper nouns. What are some common POS that occur around proper nouns?

Is there a general rule/convention for this?

Improve this question
2
  • Proper nouns often (probably usually) lack articles such as "the", "a", and "an".
    – Michael
    Commented Jan 7, 2016 at 13:51
  • 1
    I'd get a large lexicon of words. When a word in your text does not appear in the lexicon, chances are very good that it's a proper noun.
    – TimR
    Commented Jan 7, 2016 at 19:26

1 Answer 1

Reset to default
1

This sounds like a very cool project. I imagine trying to develop a program that can take a document that is entirely lowercase and determine which words within it are proper. I just think it might be doable.

Here are some characteristics that might make a word more likely to be a proper noun:

  • Ending in with 's or s' (if it's possessive or a subject, it's more likely to be proper)
  • Being preceded by Mr., Mrs., Ms., etc.
  • First word of a paragraph*
  • First word after a period*
  • First word after the first comma in a sentence, if within first five words.*
  • First letter of word starts with J (A disproportionate number of names:words start with J)
  • Word before a word that ends 't or 'll
  • Penultimate word before , or . is "to" (Means last word is an object without an article, so more likely a proper name)
  • Word ending in Y (A disproportionate number of words ending in Y are proper names)
  • Word appearing before Jr., Sr., Esq., MD., II, III, IV, etc.
  • Any single letter followed by a period.
  • Word immediately following any single letter followed by a period
  • Word immediately before any single letter followed by a period
  • Any series of two-word pairs separated by commas, along with two words before first comma and the second and third word (in order to exclude "and") after the last comma (two-word pairs in a list separated by commas would have a likelihood for being first and last names)

Using these and others in combination, you should be able to identify which words in a sentence are more and less likely to be proper nouns. For example, if a word is identified as fitting more than one of these criteria, its likelihood for being a proper noun increases. In this manner, you can begin an algorithm that essentially develops a profile for each word within a text.

Incidentally, if I were to cheat, I might be tempted to write code that imported the lists from Social Security's website of the 1,000 most popular boy's and girl's names.

*A very likely position for a subject, which subjects are more likely to be proper nouns

Improve this answer
4
  • thanks for the helpful comments. I have had most trouble with organization names like Boeing. Initially, i tried an approach involving stemming and lemmatization. but things like Boeing will be stemmed to Boe. Even if i am able to tag Boe as a proper noun, i have lost the original word
    – AbtPst
    Commented Jan 7, 2016 at 14:28
  • Boeing actually has a characteristic most English words don't have: three vowels in a row. You could come up with attributes like this that names have that standard American words don't in order to catch those that fall between the cracks. Lematizing the text is a good idea, but I would figure out a way to preserve the original text.
    – Benjamin Harman
    Commented Jan 7, 2016 at 14:46
  • Another tack you might take is to identify other keywords that appear near business names. For example, Boeing often appears with the word Company or Co. immediately after it, which could be applied more broadly. Also, the word "air" and words with "air" in it often appear in sentences with "Boeing" in it. With that in mind, you could include criteria to identify words like this, industry or tech words, that make the appearance of a business's name nearby more likely.
    – Benjamin Harman
    Commented Jan 7, 2016 at 15:02
  • bill will bill will.
    – AmI
    Commented Jan 7, 2016 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.