In Google Books search results for 1900–2008, retryable (blue line) and retriable (red line) are very similar in total number of matches:
Although retriable seems to have gained the upper hand since about 1995, its relatively strong showing in recent years may be due in large part to a house style preference in its favor at one publisher, Springer-Verlag, whose titles account for eight of the nine matches for retriable in the year 2006.
At this point, it's impossible to say that one spelling is correct and the other is incorrect—in fact, it's impossible to predict with any confidence which one will win out in the battle for dominant usage. However, the more frequent the usage becomes, the more pressure there will be on publishers (assuming that publishers in the traditional sense of the word continue to exist) to standardize on one spelling or the other. If the need to refer to something's ability to be retried continues to grow in computer terminology—and to hold steady in legal terminology—we should see some movement toward consensus on the spelling, one way or the other, in the next decade.