I don't see any change in the opposite direction. For example, one who has been retired is called a retiree (not *retirer), although it's common to describe a person as having retired. Someone given refuge is still called a refugee, not a *refuger.
Is there any objective (quantitative) evidence that this is a trend?
The dilution you are seeing is not a new set of uses in slightly different circumstances, but rather you're recognizing that the consistent usage always had a broader meaning reflecting the original meaning of the suffix in French.
The '-ee' suffix comes from the French past participle formation mostly corresponding the English past participle ending '-ed'. The past participle can be used as an adjective, or as a person with the attributes described by the adjective. That is general enough to include both the object of an action (in the pair agent/object like employer/employee from someone employs someone else; someone else is employed and is called an employee) or someone does something (the man attended the conference, so the man is an attendee).
word-forming element in legal English (and in imitation of it), representing the Anglo-French -é ending of past participles used as nouns (compare -y (3)). As these sometimes were coupled with agent nouns in -or, the two suffixes came to be used as a pair to denote the initiator and the recipient of an action.
Looking at your examples I don't see the trend you are suggesting, and every term has its own etymological development:
escapee (n.) is a more common term than escaper but it not a recent one: Ngram:
"escaped prisoner or convict," 1865, American English, from escape (v.) + -ee.
1680s, from French refugié, noun use of past participle of refugier "to take shelter, protect," from Old French refuge .............. The word meant "one seeking asylum," till 1914, when it evolved to mean "one fleeing home" (first applied in this sense to civilians in Flanders heading west to escape fighting in World War I).