It’s hard for me to guess how to pronounce words beginning with re- correctly.
Sometimes it is /rɛ/ as in reference, but sometimes it is /ri/ as in report.
Is there any rule about this?
Yes, there is a rule, and that rule is that you must look them up in a dictionary if you are not a native speaker.
That’s because words beginning with re- in English can, depending on the word, be pronounced with any of eight different vowels:
The last three or four at the end of that list tend to be for native words, while the ones at the beginning tend to be for unassimilated imports.
But in diphthongs like reindeer or reynard — let alone reiter or rearward — all bets are off.
Here’s an alphabetized sample list, with pronunciation following:
And here grouped by pronunciation:
Tchrist's answer does a good job of showing the variance in pronunciation for words starting with "re." However, there are some patterns, so I thought I'd make a post describing them, since some people might find it useful information. In particular, the pronunciation is a lot more predictable if we only look at words with the prefix re- (derived from Latin, with the general meaning of "again") rather than all words that happen to start with the letters "re" (like "reindeer" and "reiter"), and if we exclude recent borrowings from other languages like rendezvous and redondilla.
John Wells's phonetic blog has a good post that describes the usual pattern. I've copied the main bit of it below for ease of reference; keep in mind that he uses e to represent the vowel in the word dress (it means the same thing as the symbol ɛ used in tchrist's post):
When re- means ‘again’, it is pronounced ri: and stressed. So we have for example reapply, renegotiate, reconsider, all with ˌriː-. In words such as refit, rethink, rerun we get the familiar stress difference between verb and noun, but the first part is always riː. A ˈrefit, but to ˌreˈfit. So far, the rule is easy.
When re- has a vaguer meaning, it usually gets weakened and is unstressed. We have ri- or rə- (it doesn’t matter which) in words like remember, retain, remarkable, repeat. This rule is easy, too. However, if the sound immediately after the re- is a vowel, then we always have ri- (= riː- or rɪ or something intermediate). The only important example is react riˈækt and its derivatives.
Although the following words are not common, I would just like to add that prefixes like re- are also regularly pronounced with a "long" vowel before "h", as in rehydrate/rehydration, rehabilitation, and reharmonization.
Now we come to the tricky bit. If re- has the vague meaning AND IS STRESSED, AND IS FOLLOWED BY A CONSONANT SOUND, then it is pronounced re-. This type includes several well-known words: relative ˈrelətɪv, recognize, reference, relevance, and also for example recompense, replicate, resonate. (Compare relate, refer, where the re- is unstressed and weak.)
If the main stress is on the syllable after the syllable after the re- (= two syllables later), we normally get secondary stress on the re-, which therefore has a strong vowel, which again is e. So we have for example reclamation ˌrekləˈmeɪʃən (compare the verb reclaim with weak re-), recognition, recommend ˌrekəˈmend, recreation, reformation, relativity, reparation, repetition, replication, reprehensible, represent, reservation, resignation, restoration.
Types 3 and 4 are therefore the words I mean when I say “if stressed…”. They are the ones that present the greatest problem for learners of EFL.
This being English, some words are irregular and exceptional. For example, many people pronounce one or all of relaxation, resistivity, retardation in a way that violates the above rules.
These principles also apply to de- and pre-. Examples: 1. deconstruct, predetermine; preprint; 2. decide, prepare; 3. deference, preference; 4. dereliction, preparation; 5. AmE treats premature as having the explicit meaning ‘before’ (ˌpriː-); BrE treats it as having the vague meaning (ˌpre-).
Note: I believe the "violation" of the rule (for some speakers) in relaxation, resistivity, retardation is related to the presence of an unreduced vowel in the following syllable.