It has always seemed obvious to me: Three-teen and five-teen are harder to say than thirteen and fifteen. People are lazy.
The rest, laziness really doesn't make the pronunciation easier.
(And, really, it would be the same with third and fifth. Try saying, once, tooth, threeth, fourth, fiveth.)