From etymonline for whirl,
figurative sense of "confused activity" is recorded from 1550s. Colloquial sense of "tentative attempt" is attested from 1884, Amer.Eng.
It may be that the "confused activity" sense developed into the "tentative attempt" sense; etymonline does not indicate, and apparently doesn't address the question of origin of give something a whirl. But one can see that the sense "make a tentative attempt to do something" is not far from the current meaning (to try doing something one is unfamiliar with or hasn't tried before or is uncertain about). There is sourceless speculation at phrases.org.uk that the phrase arose from spinning a roulette wheel, or a "wheel-of-fortune", or a "whirlygig" colonial toy.
Also see entry in thefreedictionary, which quotes the Cambridge Dictionary of American Idioms as saying of whirl,
to try something; Usage notes: often used in the form give it a whirl: We've always wanted to take a cruise in the Caribbean, so we decided to give it a whirl.