The problem is that English uses two different kinds of adjectives to mean "first, second, etc". The ones in -ary without the -n- come from the Latin ordinals, "first, second, etc."; but they are different after 3.
- Primus — primary "first"
- Secundus — secondary "second"
- Tertius — tertiary
- Quartus — *quartary
Quintus — *quintary
The -arius suffix is also used in Latin with ordinals, and secundarius means something like "second, pertaining to two, second in rank", though it often comes very close to the simple ordinal secundus. It usually adds some connotation of ranks and order in a grand system. There is also secundanus, which I believe isn't much different.
The -n- ones come from Latin distributive adjectives, "one each, two each, etc."; they are always used in plural. They were sometimes also used in a sense roughly similar to the ordinals, which is probably why English uses them in an odd way.
- (Singuli — single "one each")
- Bini — binary "two each"
- Terni/trini — ternary/*trinary
- Quaterni — quaternary
- Quini — quinary
- Seni — senary
- Septeni — septenary
- Octoni — octonary
- Noveni — *novenary
- Deni — denary
- Undeni — *undenary
- Duodeni — duodenary
Terni/trini deni — *ternidenary/*tridenary
I believe the ones derived from ordinals were originally used to mean "second [in order]" in English, and the distributive -n- ones to mean "of two parts", or "characterised by the number 2". But then, because these meanings are related and often overlap, they got mixed up, resulting in the current defective lists, where the -n- forms serve both senses from 4 up.
The number one is the strangest exception of all, where a new word unary was made up, though no Latin equivalent exists (there is only unus, "one", but that is like using *duary from duo, "two"). Nonary is odd as well.
These are the Latin cardinal numbers for reference:
- Unus/una/unum/etc. (depending on gender and case) — "one"
- Duo/duorum/duarum/etc. (depending on case and gender) — "two"
- Tres/trium/etc. (depending on case)