The earliest idiomatic use of "island time" that a Google Books search finds is from Dewey Ganzel, "Chronology in Robinson Crusoe," in Philological Quarterly, volume 40 (1961) [combined snippets]:
Crusoe spent many days salvaging the ship and months securing his habitation on the island, and after he had "made ... a table and a chair" (p. 76) on November 12 (p. 79), he began a journal (p. 76), making entries retroactively for the period September 30–November 12, briefly recounting his labors under datelines, and making almost daily entries thereafter (with a hiatus from January 3 to April 16) until July 4 {1660} (p. 106), when his entries became more infrequent, ceasing finally, when his ink began to fail, with an entry for the first anniversary of his arrival, September 30 {1660} (p. 114). Thereafter the chronology adopts "island time," e. g., "in the third year of my being here" (p. 160), using day-month dates only rarely. Crusoe kept his calendar by means of notches on a crossed stick; this was his sole means of determining days of the month, and he discovered years after that he had lost a day or two in his account (pp. 104, 115).
This is clearly not the sense of "island time" as relaxed, unhurried, and (for some interested parties) catered that emerged somewhat later.
The next-earliest match uses the term in a more familiar way. From John Keats, "The Magic Isolation of Island Life," in Travel & Leisure, volume 4 (1974) [combined snippets]:
Fishing is both work and fun for author John Keats. He and his wife live on the top floor of the white boathouse (at left in photograph on opposite page), and Keats writes in his studio, surrounded by pines, on the other end of the island. The different beat of island time allows the writer quiet moments to read in his rocker on the boathouse porch (above).
This time, the island in question is one of the Thousand Islands in the St. Lawrence River, the waterway that (along with Lake Ontario) serves as the border between New York and Ontario.
Next, two matches for "island time" in more-or-less the modern idiomatic sense appear in 1983. From Linda Levine & Garfield Barbach, The Intimate Male (1983):
When we were meeting cross country, we'd fly in and have forty-eight hours with no other agenda except us. We called it "island time" because it felt like being off on an island together. ... Now, in order to recreate that, we go away for at least one weekend out of a month. Or we create "island time" in our own home.
From "Planning for the Virgins" in Yachting (February 1983):
Whenever I'm asked what someone should pack for their week's cruise in the V.I. I answer, "Not much." It's difficult to explain the lifestyle adopted once you get on "island time" and what few things you really need. Casual, casual, casual and light is the best way to go. Days are spent primarily in bathing suits and tee-shirts with the occasional addition of shorts and shoes for shoreside visits.
By 1985, Port Aransas, Texas, was using "Living [or Live] on Island Time" as a slogan in advertisements in the March 1985 and June 1985 issues of Texas Monthly magazine. Elvi Whitaker, The Mainland Haole: The White Experience in Hawaii (1986) offers multiple synonymous terms for the difference in pace of island life in Hawaii:
The relative slowness of Hawaiian time is captured in such mainland concepts as "island slows," "Maui slows," "pineapple time," "island time," "Mexican time," or even "mañana." The reference to Mexican time is probably to be expected as the actual experience of most migrants to Hawaii has usually not stretched beyond the boundaries of the continental United States. Here any difference in time code, particularly a slower pace, is labeled, somewhat pejoratively, in recognition of the southern neighbor.
It's worth noting that "island time" lurks in a number of very old narratives, including The Odyssey, where time slows down—or speeds up, depending on how you look at it—at such stops as Calypso's island, the island of the Lotus Eaters, and Circe's island. Another interesting legendary island is discussed in H.P.A. Oskamp, The Voyage of Máel Dúin: A Study in Early Irish Voyage (1970):
Both the episode of the Queen and her Seventeen Daughters in M[áel] D[úin], and its parallel in Bran (the Island of the Women, 62-63) consist of three elements: the island is inhabited by women; on the island time does not pass as in the natural world; the leader of the women wields a clew which clings on the hand of the man who catches it.
So it seems that people have been aware of the intoxicating and time-retarding charms of islands for many centuries, though they may not have had recourse to the term "island time" as a way to characterize until the late twentieth century.