The suffix comes from the latin suffix -ivus (pronounced [iːwus]). In French, Latin declensions gradually eroded, so the suffix became -ive, ending with a [v] sound (a common evolution from the Latin [w]) and a schwa (a common evolution from Latin declensions). The schwa further disappeared from the masculine form, and the final /v/ sound changed to [f], I think under German influence.
English acquired the suffix via Anglo-Norman French in the 13th and 14th century, in the masculine form: actif, motif, etc. Later, in the 15th and 16th century, the suffix was added to other words directly in English (some of which then made their way back to French, such as competitive and sportive). As these words were usually created by scholars who knew Latin, they used a suffix that was closer to the Latin original, thus -ive. Over time the older -if forms evolved to match the expanding -ive forms.
The few -if words in modern English such as motif, aperitif, etc. (and a few others that aren't from French) are 19th century additions, not survivors from Anglo-Norman, except waif (which is not constructed with an -if suffix) and perhaps naif (I don't know whether this one is a survivance or a later reintroduction).
Source: Latin Suffixal Derivatives in English: and Their Indo-European Ancestry by D. Gary Miller (2006, Oxford University Press), cited in Wiktionary.