Apparently the cocoa-nut shy 'had nearly superseded the more ancient "cock shies"' by 1884:
The 1787 citation from the Online Etymology Dictionary is from Jeremy Bentham's Defence of Usury:
All that he knows, all
that he can know, is, that the enterprize is a project, which,
merely because it is susceptible of that obnoxious name, he looks
upon as a sort of cock, for him, in childish wantonness, to shie
One 1815 account says normally "a snuff-box, a slice of ginger-bread, or a penny-piece will suffice", but describes the brutal game of a group of sailors in France throwing stones at a live cock on top of a six foot pole.
Apparently it was an ancient Shrovetide custom to play games outside, and one such game was putting a cock on a pole and killing them with thrown short sticks. From 1837:
Shrovetide was in these countries long infamous for the barbarous massacre of cocks, which, at that season, were fastened to a stake, and thrown at with cudgels till their limbs were broken in detail (as, of old, those of the felon were on the wheel), and their bodies "reduced to a jelly." This custom has given origin to a term sacred in the school-boy jargon of old England, where every object set up as a mark for a stone is pueriliter vocatus " a cock-shy." It is not, perhaps, impossible that from this ancient usage was derived the custom of calling together the Parliament sometime about Shrovetide, on which occasion his majesty's minister for the time being is set up to be flung at by his majesty's opposition usque ad delicias Votorum; and thus, the said minister, though seldom entitled to be considered as a shy cock, is made a cock-shy to all the malcontents in the kingdom.
It's not certain, but this possibly links a timid, scared, shy cock to the act of throwing and the coconut shy. In any case, all the verb meanings of the word I found are in connection to throwing at cocks, as this game was also known. (Hackwood's 1907 Old English Sports has a whole chapter on its history and its links to Shrove Tuesday.)
From The Critical Review, 1780 (excerpting Account of a Debate in Coachmaker's Hall by Harum Skarum Esq., also 1780):
... but me'thinks I knows about shying at cocks as much as he. I remember what monstrous good fun we had last Shrove Tuesday in our lane. My heart! what fun it was! Why we shy'd down all the cocks in the neighbourhood.
... I do love shying at cocks, that I do, because it is such fun. And what business has the Pope with my cocks ? I'll shy at 'em for all him, that I will."
A New Review, 1782:
I have been with Dick Brisk shying at his cock.
The verb shy definition in The Routledge Dictionary of Historical Slang by Eric Partridge (1973) quotes Bentham's 1787 and says it's perhaps an example of shy(-)cock, itself defined:
shy(-)cock. A wary person, esp. one who keeps indoors to avoid the bailiffs: 1768, Goldsmith. 2. Hence, a cowardly person: 1796, F. Reynolds. Both senses by ca 1850, the latest record being of 1828. Prob. ex lit. sense, a cock not easily caught, one that will not fight.
(The first meaning can be found in Grose's 1785 A Classical Dictionary of the Vulgar Tongue.)
Finally, John Ayto's 1990 Dictionary of Word Origins says:
It is generally assumed that shy 'throw' [18th century] must have come from it [the 'timid', 'reserved' meaning], but the exact nature of the relationship between the two words is not clear. The original application of the verb seems to have been specifically to the throwing of sticks at chicken and it has been suggested, not altogether convincingly, that its use alludes to the notion of a 'shy' cockerel that refusese to fight (there was an 18th- and early 19th-century slang term shy-cock which meant cowardly person.