1

I'm working with football (the European variant) commentary, looking at word frequency. Often enough, the commentary will spell their excitement by writing 'Gooaal', which for the purposes of my word frequency is different from the properly spelled 'goal'. I'm considering capturing these cases by removing all successive duplicate letters. This question is not about how to do that, but I need a guess of how big an error will I create with this technique.

How many words collapse to the same word (string) if one remove duplicate letters? I need a just rough estimate. My guess is less than 1‰ but I'm not a native speaker so I may miss cases.

To make it clearer, here is the first paragraph of this question after running it through the algorithm:

I'm working with fotbal (the European variant) comentary, loking at word frequency. Often enough, the comentary wil spel their excitement by writing 'Goal', which for the purposes of my word frequency is diferent from the properly speled 'goal'. I'm considering capturing these cases by removing al sucesive duplicate leters. This question is not about how to do that, but I ned a gues of how big an eror wil I create with this technique

(I have no idea what tags are appropriate, please add more relevant ones.)

Improve this question
2
  • In Hungarian tizenegyedik and tizennegyedik have distinct meanings: one means 11th, the other 14th.
    – bof
    Commented Apr 26, 2018 at 9:49
  • Youre gona create a huge boatload of erors.
    – Arm the good guys in America
    Commented Apr 26, 2018 at 14:37

2 Answers 2

Reset to default
3

It would seem more efficient to write an algorithm that would capture any number of repeated o’s and a’s to capture these variations on “Goal!” than to eliminate all double letters in your entire database.

The frequently used verbs meet, speed, bleed, and feed form their past tenses by shortening the vowel to met, sped, bled, and fed.

Other minimal pairs that will reduce to the same result:

refereed, referred
feel, fell
devotee, devotees; devote, devotes
divorcee, divorcees; divorce, divorces
stooped, stopped
steeped, stepped
career, carer
kneel, knell
pall, pal
teen, ten
commandeer, commander

There are simply too many common words your program wouldn’t distinguish. Even if you had to deal with “Goooaaal!” and “Gooaal!” separately, it would seem a better use of your time — and risk less skewed results — if you abandoned this method of sorting your data.

Improve this answer
1
  • I'm sure there's a regex expression that would be able to capture all possible (mis)spellings of "goal".
    – Pam
    Commented Apr 26, 2018 at 13:00
2

In English there are many words which will collapse to the same string, e.g. to/too, of/off, son/soon, desert/dessert, princes/princess.

Improve this answer
1
  • +1 Small simple common words like to/too and of/off will definitely appear frequently in commentary.
    – AndyT
    Commented Apr 26, 2018 at 13:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.