except the gender inflection in words like fiancé and fiancée.
Would the reason lie in the fact that maybe these words have not been around for a long period of time so they kept their original distinction or would there be other factors?
For a start, the degree to which loan words will be inflected according to the rules of English, the language borrowed from, or a hyper-correction which mistakenly applies a foreign ending is a flexible matter. We can find all three being current such as for example octopuses (English plural rule), octopodes (correct Greek plural rule) and octopi (a common but double mistake, trying to apply a Latin plural rule, and even then the more reasonable plural would probably be octopedes).
The general trend is to treat words very much as foreign words to begin with (including the borrowing of the foreign plural, foreign forms for other genders, etc.), more and more as English words as time goes on, though some features often last a long time especially if those familiar with the word are familiar with the other feature.
This makes it difficult to predict how long a word will take to move to being fully English in style. We can guess that an unfamiliar word will remain "foreign". We can guess that a word where the foreign feature is well-known to remain foreign. We can guess that a word whose foreign feature is unusual to remain foreign, such as how par excellence comes after the noun rather than before (but note that galore also comes after the noun despite having mutated from go leór in spelling and meaning - it's a different word only in English and no longer a loan word at all, but it retains a feature from Irish).
But we can do no more than guess. The rate of assimilation is never fully predictable.
Now, all that said, in English gender is mostly a covert category. Pronouns have a gender, but it refers to the gender of the referent, not to the word. Hence I might saw "she is a good actress" but I might also say "she is a good performer"; my choice of she did not relate to the gender of the word actress but to the person in question.
The nouns we have that reflect gender will either be specific within a category (actress is an actor who is female, bitch a dog who is female, drake a duck who is male), or to assumptions about gender (when fireman was coined it was assumed that women would not do such a job) or else have been coloured by such (etymologically there is nothing female about midwife as a man doing midwifery is still "with a woman" as the word describes, but the word is so coloured by assumptions).
There is also a general move within English to remove gender where it is not useful (pretty much everywhere except relating to biology) as after all to anyone not casting a role the distinction between "actor" and "actress" has had no real relevance to anything since the days when the profession overlapped heavily with that of prostitution.
French has a completely different gender system where while gender will often equate with sex when it comes to people or sex-specific terms for animals, there were be a general correspondence of perceived gender and grammatical gender, but there would also be a masculine or feminine language gender where the perceived gender is neuter (only the most Freudianly inclined would consider a pen to be masculine).
As such, most borrowings of nouns from French immediately lose their gender on being imported into English, unless they are either borrowed as a phrase or there is an English phrase created from borrowed words (not all all-foreign phrases are single borrowings, "nom de plume" for example is an English coinage out of French words for where the French would have used "nom de guerre").
Most importantly, most phrases that would use such a word would have little use for the feature of gender - it isn't going to affect anything else about how the word is used, since the language borrowed into has no need to e.g. match pronouns or adverb forms.
Now, English does, as said, gender some words that refer to people. One of these words is husband (in most senses of the word, including that as relates to marriage), and another is wife. This is all the more explicit since we've just moved out of a period where marriages where always of a man and a woman*.
And so, it fits into how English works, that fiancé or fiance is a person who will become a husband, and fiancée or fiancee is a person who will become a wife.
About the one pressure on the word is that the pressure to drop diacritics leaves the form fiancee where a double unpronounced E is strange, which could lead it to then become fiance. This has yet to happen, if it ever will, and the accented form seems to still be the most common. (Notably, my spell-check is complaining about the unaccented form, but not the accented).
*Same-sex marriage was banned in Rome in the 4th century and this prohibition moved out throughout Europe with the spread of Christianity, while there were no recognised same-sex unions in the West until the 1980s, no same-sex marriage until the 1990s, and some countries still don't have same-sex marriage. While the redefinition of marriage as different-sex-only is being reverted back again, marriage has been different-sex only for the whole period that the English language was formed.