From my experience, it seems that although unstable is more commonly used, instable is often preferred in engineering and scientific contexts, e.g. "aircraft instability", "instable algorithm".
Are there any differences in the implied meaning of the two terms? Should unstable be preferred?