According to the Oxford English Dictionary1, the word entered the English lexicon (as a direct French borrowing) way back in the mid-1500s, at a time when spellings were far from standardized, on either side of the channel.
In the OED's earliest attestations, from 1553, it is actually spelled portemantew, retaining the -e on port- and using a, perhaps, slightly more phonetic spelling of the last syllable. This was far from the only spelling available, however. Other early spellings include portmantew (1598), port-manteawe (1635), and portmantoe (1689).
The word's pronunciation wasn't regular in the early centuries of its use, either. It also took different forms (often with some variation in spelling, as well), most regularly portmantua (first attested in 1581 and most recently in 1852, apparently from a regular confusion with the cloth made in Mantua) and portmantle (1602-1924). Some less-common versions include porte-manque (*a*1613), Port-mantick (*a*1670), portmante (1680), portmanty (1897), portmantuan (1632), Portmantium/Portmanteam (1682, both spellings used in the same work), and Portmanten (1698).
The first attestation for the modern spelling of portmanteau is attested as early as 1635. This spelling seems to have regularized sometime in the mid-to-late 1800s, perhaps helped along by Lewis Carrol's famous usage in 1871.
As to where the -e from porte went? You may notice that a common feature of the majority of the spellings listed above is the absence of said-e. English spelling is not subject to hard-and-fast rules, but it may be that it was omitted because the syllable is said just like the English word port, or perhaps because its inclusion suggests an extra syllable that is not intended2.
1 "portmanteau, n." OED Online. Oxford University Press, March 2017. (Behind a paywall, unfortunately; if you don't have access, check with your local library, which may have a subscription.)
2 The -eau ending is not as confusing as you might guess, as it is virtually always pronounced like oh; it does sometimes shout "this word is French!", as in chapeau or eau de cologne, but it also is used in a few everyday words that we don't much think of as borrowed anymore, like bureau and plateau.