I am working on an app to help people learn English as a second language. I have validated that sentences help in learning a language by providing extra context. I did that by conducting a small research in a classroom of 60 students.

I have mined over hundred thousand sentences from Wikipedia for various English words (Including Barrons'800 words and 1000 most common English words)

Entire data is available at https://buildmyvocab.in

In order to maintain the quality of content, I filtered out sentences which were longer than 160 characters since they might be difficult to understand.

As a next step, I want to be able to automate the process of sorting this content in the order of ease of understanding. I myself am a non-native English speaker. I want to know what features I can use to separate easy sentences from difficult ones.

Also, do you think this is possible? As English experts do you think we can make a machine understand the complexity of sentences?

  • There are probably about ten different strategies one might use for this, all of them deficient in some regard. The first step in most strategies would be to parse the sentence into a syntax tree. Then one can apply several different measures of tree complexity, including, notably, elisions and various types of indirect references. (I'm pretty certain this has been done a number of times already, but I can't give you any references -- the last I read about was probably 20 years ago, in CACM or some such.)
    – Hot Licks
    Jun 3 '17 at 21:55
  • 1
    NLP is a massive and complex field of computer science, and of the software industry around us. The world's major nlp products (i.e., one may have heard of "Google", there's another small company called "Apple" tackling it) are enormous enterprises with vast numbers of scientists and theoreticians (not to mention legions of toiler mere engineers) working on the systems, such as Alexa, etc. Really, this question is much like asking "I'm interested in building a 'suspension bridge', where do you buy the wire and how thick should it be?" It's just not sensible.
    – Fattie
    Jun 4 '17 at 11:43

Purely FWIW, if you're a hobbyist programmer:

I direct you to any of the service platforms which are the expression of the world's cutting edge in NLP, which is a massive and evolved industry. (Indeed it is, and is becoming more so, one of the world's biggest technologies - who hasn't used voice input by 6am these days?)



The Natural Language Classifier service understands the intent behind text and returns a corresponding classification, complete with a confidence score. For example “What is the weather like today? or “Is it hot out?” or “Is it going to be nice today?” are all ways of asking about “temperature”. Use NLC to answer questions in a contact center, create chatbots, categorize volumes of written content and more.



Amazon AI services bring natural language understanding (NLU), automatic speech recognition (ASR), visual search and image recognition, text-to-speech (TTS), and machine learning (ML) technologies within the reach of every developer. Based on the same proven, highly scalable products and services built by the thousands of deep learning and machine learning experts across Amazon, Amazon AI services provide high-quality, high-accuracy AI capabilities that are scalable and cost-effective...

The other two of the big four being obviously Google and Apple.

You can go ahead and use these completely free as a hobbyist, enjoy.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.