Take the 2-minute tour ×
English Language & Usage Stack Exchange is a question and answer site for linguists, etymologists, and serious English language enthusiasts. It's 100% free, no registration required.

I'm in the process of creating a parser which, using a few dictionaries (English language words, places and acronyms), splits the domain name into a set of potential phrases and attempts to decide which phrase is the most likely to be correct according to the intentions of the person who registered the domain.

My current rules are:

  • The place name must be at least 4 characters long.
  • A maximum of 2 place names can occur in a phrase and must be adjacent
  • Acronyms must be at least 3 characters in length
  • Priority (in order) is:
    1. least number of words
    2. highest average character length
    3. highest maximum word length

For example, refundslondon extracts to refunds london, refunds lon don and refunds lond on.

It doesn't seem to work too badly at the moment, but from an English-language perspective, it would be nice to know if there was a way to determine (a) the validity of the sentence structure and (b) if certain words appear out of place, particular with respect to words that technically can't really go on the end of a sentence.

Examples:

  • my probably shouldn't go at the end of a sentence
  • and probably shouldn't go at the start or end of a sentence

...and so forth.

Any assistance would be appreciated!

share|improve this question
2  
You'd be surprised how difficult it is to make hard and fast rules like these. It might even make you say, "oh my!" And why, exactly, should and not go at the start of a sentence? :) –  Kosmonaut Oct 25 '10 at 2:44
    
You are right in a broader sense, but it's easier to generalise when limiting the context to something such as domain names. –  Nathan Ridley Oct 25 '10 at 2:51
    
You think so? I would think that domain names would be more likely to stretch the limits of usage than other contexts. But at least the domain names are going to be made up of a very limited number of words on average; that is definitely in your favor. –  Kosmonaut Oct 25 '10 at 3:19
1  
That's a challenging task. I am not quite sure how a parser could possibly deduce "the intentions of the person who registered the domain" from the domain name alone, without parsing at least the home page as well. There are lots of domain names where even humans have trouble figuring out the correct reading, you probably already know the most famous examples: whorepresents, expertsexchange, penisland, therapistfinder, molestationnursery, powergenitalia... Your rules would not work for the last two. –  RegDwigнt Oct 25 '10 at 10:06
4  
This question would actually be more appropriate for StackOverflow or Theoretical Computer Science even! –  Noldorin Dec 3 '10 at 23:12

1 Answer 1

Tag the words with their parts of speech. Use a Markov model to see which parts of speech are most likely to follow other parts of speech (NOUN - VERB - ADVERB is probably more likely than ADVERB - NOUN - VERB). Use the probabilities to help determine which sentence is correct.

share|improve this answer
    
For best results, the statistical probabilities of tag sequences should be gathered by examining a large collection of manually split/labeled website names. With almost every project of Natural Language Processing, greater accuracy is achieved by tailoring the process to the problem set. –  Xantix Dec 13 '12 at 5:29

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.