1

Backstory (You can skip):

I am programming a word parser that will do things like detect acronymns and abbreviations, which will commonly have periods appended to them. The goal is to determine however when a period is meant to denote an acronymn/abbreviation, and when it is denoting a sentence ending.

Here is an example I am working with:

🞂 In 1968 Rover and its Alvis subsidiary were incorporated into the Leyland Motor Corporation later British Leyland or BL. In 1981 the then nationalised BL sold the Alvis business to United Scientific Holdings for £27 million.

You might think this is a trivial problem that can be solved via regular expressions. One for example could just look for a proper word or an uppercase letter following it, but I assure you that it starts becoming complex. Take these snippets which have previously tripped me up:

arrived c. 1778, at which time ...

born approx. July 1941, when...

The Panzer II Ausf. A to C had 14 mm of slightly sloped...

None of those were sentence breaks, and were furthermore, followed by capitalized words. There is a lot to actually consider, and its not a problem with a perfect solution.

Anyway, In my existing code, I have been making an assumption that words that are comprised only of Latin Consonants denote an abbreviation, and not a sentence break, such as BL.

In this case, it is both an abbreviation and a sentence break.

If I had a list of sentence starting words, I could check what follows and be able to increase my accuracy.

Actual Question:

What are the most common English words to start a sentence? Thanks.

  • 1
    Interesting. But ANY word can start a sentence. So you're just looking for the commonest words in English aren't you? – Old Brixtonian Jun 21 at 6:57
  • Python's NLTK can break text into sentences for you. It's going to be much easier using a tool built for this than trying to roll your own. More here: pythonspot.com/tokenizing-words-and-sentences-with-nltk – Benjamin Jun 21 at 7:11
  • @OldBrixtonian I was told, words like, And and But should not start words, hint hint ;-). I also can't imagine how you would start a word with ate or breathed. – Akiva Jun 23 at 0:53
  • @Akiva Rules like that are for children really. Grown-up books don't bother with them. (Perhaps the fear of starting a sentence with And is why Sarah Palin keeps using Also: it's a hypercorrection.) In the King James Bible, Genesis 1 (The first book) has 31 verses, 29 of which start with And, while 2 Kings 5 has four verses starting with But and fifteen with And. – Old Brixtonian Jun 24 at 10:42
  • 1
    @Akiva Starting sentences with Ate and Breathed: "'Ate what?' she asked." "'Breathed heavily, did he?'" "Got up feeling lousy. Ate rat poison. Felt ill. Breathed my last." "Ate and ate and ate. That's all they ever did." (That last narrator's English isn't great.) – Old Brixtonian Jun 24 at 11:01
1

You can use Python's NLTK tokenizer to split text into sentences. It has a good list of abbreviations to handle sentences like "arrived c. 1778, at which time ..." but you will have to add some abbreviations to it as well.

import nltk
nltk.download('punkt')    

abbreviations = ['approx', 'ausf']
sentence_tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
sentence_tokenizer._params.abbrev_types.update(abbreviations)
sentence_tokenizer.tokenize("He arrived c. 1778, at which time he left. He was born approx. July 1941, when he died. The Panzer II Ausf. A to C had 14 mm of slightly sloped.")
  • I actually already do something like that (A lot more complex actually), but do you have a list of those abbreviations? That could be useful. – Akiva Jun 23 at 0:52
  • 1
    Just type: print(sentence_tokenizer._params.abbrev_types) – Benjamin Jun 23 at 0:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.