I found out that in fact there are many words that have the pure "e" sound. A few examples: back, cash, cat, fact, relax, track, crash.
No, they do not. The IPA /e/ sound does not occur in those words. You have confused it for the /æ/ vowel, which does occur in them.
Listen to these Youglish samples for fact and for relax, and try hard to find even one that has an /e/ in it for the given word. Use the "skip-forwards" button that looks a little like ⏭ to move through the thousands of samples, or the "skip-backwards" button that looks a little like ⏮ to move back through the list. You can do the same with the other words you mention.
Where are you thinking you're hearing these? Very few regional accents pronounce the vowel of ash /æʃ/ with any kind of an "e" sound, although a few do. Are you in one of those regions?
If you do not have all three of /e/ and /ɛ/ and /æ/ sounds in your native language as contrasting phonemes, then you may not be hearing them correctly. If so, then you may not be able hear the clear difference between triples like mate with /e/, met with /ɛ/, and mat with /æ/.
Are those three different phonemes for you? What's your first language? I ask so we can look at its phonemic repertoire, the aural lens through which you may tend to hear the world.
You earlier wrote:
For me as a foreigner it is impossible to distinguish the pronunciation of "set" versus "sat". Are native English speakers able to hear a subtle difference between the two?
This shows that you cannot hear those sounds correctly. To us, they are worlds apart, just as meet and mitt and met and mat and mate all are.
One last thing about dictionaries. They never have real pronunciations. So they will always be inaccurate. That's in part because to a large extent, saying that there is one "American pronunciation" or there is one "British pronunciation" is a myth. Better to look at each region for hand, lamb, bath. Each of those has dozens of different regional pronunciations by native speakers. No dictionary ever covers real pronunciations to this degree of accuracy because it will never help anyone to do so. So the idea that the dictionaries haven't caught up to reality is ill-founded: they never shall.