3

I am currently trying to build a database of English words and their hyphenations (end-of-line divisions) (en-US, if it matters), and thereby have come across some words which I have found contradicting hyphenations for. If those words were exotic, I would not be wondering about it, but some of them are frequently used. For example:

  • Germany: Merriam-Webster - Ger-ma-ny; Hunspell (which by far is the most dominant spell checker and hyphenator in the open source scene, driving applications like LibreOffice, OpenOffice, Firefox, Thunderbird and the like) - Ger-many

  • freely: Merriam-Webster - free-ly; Hunspell - freely

  • rapid: Merriam-Webster - rap-id; Hunspell - rapid

I have read a lot of articles (most of them on this site) about hyphenation. The general consensus seems to be that we should look up the respective word and its hyphenation in authoritative sources. But what if those sources contradict each other?

Another advice which often was given was that we just should hyphenate between syllables. Since I am not a native English speaker, this is extremely difficult for me. While I would have done it right with Germany and freely, I would never have done it right with rapid (in my world, it would have been ra-pid).

I always have considered the Oxford English Dictionary to be the most authoritative English dictionary. Imagine my surprise when I saw that they neither show hyphenation nor syllabication. The Wiktionary does show hyphenation, but only for some words; the examples mentioned above, being very common words, are not among them, so it's worthless in this respect.

Could somebody please give me a hint what I should do if two important sources which both can (somehow) be considered authoritative show contradicting hyphenations, and even more important, could somebody please tell me if there is a reliable method to identify words which are suspect in this respect in the first place?

To explain the latter: I am currently using the hunspell data to build my database semi-automatically; otherwise, I couldn't handle it. The hunspell data is the only one I have found to be usable to get the hyphenation of a word quite easily.

As a second step, I would like to be able to identify and separate suspect words, which I then could look up manually in different sources (hoping that only about 5% of the words are suspect).

EDIT 1

As a reaction to one of the comments, I now have found a word where at least 3 characters are left at each side after hyphenation, but where different "authorities" hyphenate differently:

Microsoft Word 2010 hyphenates inconceivable as in-con-ceiv-a-ble, where Merriam-Webster has in-con-ceiv-able.

Another one: Merriam-Webster says cli-ent, where hunspell says client, i.e. does not hyphenate that word at all.

EDIT 2

@Hot Licks has pointed out that the dictionaries are showing syllable boundaries, not hyphenation points (if any). However, at least in case of Merriam-Webster, this is the same. From their dictionary API documentation:

<hw>...</hw>    (text = boldface)
    HEADWORD
    - This is the first bold word in an entry
    - contains "syllable" break points (that is, 
      end-of-line hyphenation points) here indicated 
      by asterisks, which will translate to raised dot, 
      {point} in Merriam-Webster font. 
    - may contain superscript homograph numbers 
      {h,1}, {h,2}, etc., in the same font (bold)
    - single word space after <hw> field

Please note the text following the second hyphen. IMHO, that means that each syllable boundary is a hyphenation point, and vice versa.

EDIT 3

I have found more precise information. From Merriam-Webster's guide to pronunciation:

Hyphens are used to separate syllables in pronunciation transcriptions. [...]

The centered dots in boldface entry words indicate potential end-of-line division points and not syllabication. [...] As a result, the hyphens indicating syllable breaks and the centered dots indicating end-of-line division often do not fall in the same places.

  • 1
    Generally speaking, you should never hyphenate a word and leave fewer than 3 characters on either side. – Hot Licks Aug 4 '18 at 11:38
  • 2
    What the dictionaries show is not the hyphenation points but the syllable boundaries. – Hot Licks Aug 4 '18 at 12:00
  • 1
    Please correct me if I am wrong, but Merriam-Webster seems to show the hyphenations. For example, consider merriam-webster.com/dictionary/calculation. Directly under the title (in giant letters), there are three entries. The left denotes the type of the word (in this case, noun), the second is what I have considered to be the hyphenation, and the third is the pronunciation, and I always thought the syllable boundaries are part of the pronunciation. Please correct me if I am wrong (which may very well be the case). – Binarus Aug 4 '18 at 12:11
  • 1
    MW is showing the syllable boundaries. Generally, hyphenation occurs on syllable boundaries, but there are limits as to which boundaries can be used – Hot Licks Aug 4 '18 at 12:29
  • 2
    @Hot Licks Please forgive me, but it seems you are wrong regarding what the dictionaries show, at least in the case of MW. Please take a look at my EDIT 3. – Binarus Aug 4 '18 at 14:08
2

The first thing you have to understand is that hyphenation in English is done on different principles:

  • An "American" system, which derives from this "Hyphens are used to separate syllables in pronunciation transcriptions." This involves two basic fallacies: pronunciation transcriptions are a rare special case of the use of hyphenation, which is normally used for texts that are to be read, not recited; and even if you wanted to use this as a base, there are lots of differences in syllabication between regional dialects.

  • A "British" system, which breaks words according to their etymological components (prefixes and suffixes etc.). This makes the word breaks easier to follow, and should be preferred. Thus: con-ceiv-able. But this puts you in conflict with Microsoft and the like, of course.

  • Thank you very much and +1 for bringing the two systems to my attention. – Binarus Apr 13 at 13:14
1

If you search hunspell hyphenation you should find an end-of-line hyphenation library (import from TeX) that should suit your needs. The min right and left lengths are variables.

I don't know if this can detect part-of-speech such as (verb) pro-ject vs (noun) proj-ect.

  • I don't believe this answers the question, since Hunspell is already mentioned in the question. – Laurel Nov 2 '18 at 3:10
  • Hunspell was being used with partial success and I'm suggesting an add-on that should complete the task, rather than re-doing everything from scratch. – AmI Nov 2 '18 at 3:20
  • @Aml In fact, this is what I currently use for the automatic part. The problem is that Hunspell's hyphenation differs a lot from Merriam-Webster's, for example. Hence, whenever I feel that Hunspell may have missed a hyphenation point, I manually look the word up in other dictionaries. This is still quite painful, but better than nothing. The most difficult part (for me) is to determine if Hunspell might have missed something, so I sometimes end up unnecessarily looking up 10 words in a row manually just because I can't trust Hunspell completely ... What a pity that MW does not offer an API. – Binarus Nov 20 '18 at 8:30
  • I'm sorry -- I didn't realize that you were already using hyphen.tex. Because it is rule based rather than a full dictionary, it can't reliably handle breaks leaving less than 3 letters. It does have an exception list at the end where you can add on, but change the file name if you customize it. You could also build hyph_en_US.dic and customize that. – AmI Nov 20 '18 at 18:34
  • 1
    @Aml Thanks ... What I did: I installed Hunspell in Cygwin. It includes an executable "example.exe" which I directly use to look up hyphenations based on hyph_en_US.dic. I didn't know that you can customize the latter; I don't need this because each word goes into a database anyway. Hunspell / example.exe has a configuration file where you can tell it how many letters must remain at each side after hyphenation, but that didn't work. Hence, I have changed the threshold from three to two letters in the source code and compiled myself, which works reliably. But still great differences to MW ... – Binarus Dec 2 '18 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.