It has always seemed obvious to me: Three-teen and five-teen are harder to say than thirteen and fifteen. People are lazy. 

The rest, laziness really doesn't make the pronunciation easier. 

(And, really, it would be the same with third and fifth. Try saying, once, tooth, threeth, fourth, fiveth.)

(afterthought)

Some people will note the middle English "fifthe", and object, but I'll point out that the cardinal is voiced as far back as middle English. You have to go back to old English to get an unvoiced "fif" as the cardinal number, and then you have to ask why middle English became voiced only in the cardinal.

It is true that the obvious always assumes the current context, but the current context is derived from the old context, and the principle of laziness applies in the old context at least as much as it applies now.

(end afterthought)