Backstory (You can skip):
I am programming a word parser that will do things like detect acronymns and abbreviations, which will commonly have periods appended to them. The goal is to determine however when a period is meant to denote an acronymn/abbreviation, and when it is denoting a sentence ending.
Here is an example I am working with:
🞂 In 1968 Rover and its Alvis subsidiary were incorporated into the Leyland Motor Corporation later British Leyland or BL. In 1981 the then nationalised BL sold the Alvis business to United Scientific Holdings for £27 million.
You might think this is a trivial problem that can be solved via regular expressions. One for example could just look for a proper word or an uppercase letter following it, but I assure you that it starts becoming complex. Take these snippets which have previously tripped me up:
arrived c. 1778, at which time ...
born approx. July 1941, when...
The Panzer II Ausf. A to C had 14 mm of slightly sloped...
None of those were sentence breaks, and were furthermore, followed by capitalized words. There is a lot to actually consider, and its not a problem with a perfect solution.
Anyway, In my existing code, I have been making an assumption that words that are comprised only of Latin Consonants denote an abbreviation, and not a sentence break, such as
In this case, it is both an abbreviation and a sentence break.
If I had a list of sentence starting words, I could check what follows and be able to increase my accuracy.
What are the most common English words to start a sentence? Thanks.