There's two things that account for most of the trouble:
The Great Vowel Shift.
The Great Vowel Shift caused the pronunciation of English long vowels to change, and many of them to become diphthongs. This is discussed in great detail in the Wikipedia article, including some nice charts. As a result, many English written vowels are not pronounced as you might expect--although the pronunciation of vowels affected by the shift is actually very regular, so long as you don't judge them by the standards of other European orthographies.
The Norman invasion
The Normans occupied England for several centuries and introduced thousands of French words into English vocabulary. The problem is that these words tended to be spelled according to French spelling conventions, which were very different from Germanic spelling conventions. This created two different, inconsistent spelling systems within the same language.
To these two big factors, we add two more which afflict the language to this day:
Extreme conservativism
English standards tend to maintain old spellings that represent the original pronunciation of a word, even if the pronunciation has changed. This is why we have a gh in cough, through, bought, etc. Even worse, sometimes English words are spelled in a way that's supposed to reflect etymology, even if the etymology is wrong. This is why we have a b in debt.
Foreign spellings
Words borrowed from other languages into English tend to keep their spelling from the source language, even if the pronunciation goes against English rules. This is why we have rendezvous pronounced, roughly, "ronday-voo", which is from French and follows the French spelling.