So, I have always pronounced rectilinear with a long i sound (recteelinear), until the other day when, much to my embarrassment, I heard it pronounced with a short i (rectuhlinear). How would I prevent this pronunciation mistake in the future? What rules are there regarding the long/short i in cases like rectilinear vs rectify?
Well, the official answer to "How do I prevent mispronunciation?" is to check to see how things are pronounced using either a dictionary (online dictionaries often allow you to simply play an audio file of the word rather than figuring out the phonetics) or Youglish. That seems a lot of effort for every word.
Given that English is spoken widely, often as a second or "official" language, and has a variety of accents and therefore a variety of valid pronunciations (and these also change gradually over time), I wouldn't feel any shame in mispronouncing words. Take the lesson, not the shame. Many people read words on a page and never hear them said before they pronounce them, too, so even native speakers have often been known to mispronounce words (like "hyperbole"!). It might even be that your pronunciation becomes the mainstream pronunciation in time.
With regards to a "rule", The free dictionary lets you look up words containing a specific combination of characters. As you can see in the link, I searched for the combination "ecti". Discarding any "tio" combinations, it seems more common to pronounce these four letters with a short "i" sound:
retrospective, rectify, effective, directive, pectin
In fact, if you narrow down the search to "recti", I cannot see a word that I would pronounce with an "ee" sound. So for "recti", I would say the rule is short "i" and would welcome any words that contravene this in the comments!