'Vicious circle' as an error in reasoning
The earliest match for "vicious [or rather, vitious] circle" that I've been able to find appears in Edward Maihew, A Treatise of the Groundes of the Old and Newe Religion (1608). Unfortunately, I haven't been able to gain access to the contents of this book beyond its table of contents, so I can't provide the actual quotation here.
However, the term appears multiple times just seven years later in John Ainsworth & Henry Ainsworth, The Trying Out of the Truth Begunn and Prosequuted in Certayn Letters and Passages Between Iohn Aynsworth and Henry Aynsworth; the One Pleading for, the Other Against the Present Religion of the Church of Rome (1615):
Now wheras you object that I know onely this promise by Mat. 16. & that by the Popes & churches sentence I know onely S. Matthewes gospell to be canonicall, and that the gospell of Nicodemus is not authenticke, I grant all, but I deny that here there is any maze or circle, that you would fayne from hence inferr, since this mutuall reference, and reciprocall dependence is in diverse kindes; and then Aristotle will tell you, that it is no circle or vitious argumentation to demonstrate a causa ad effectum et ab effectu ad causam; and a younge Philosopher wil tell you that the materia and the form doe mutually depend, and reciprocally cause one an other, but the one in genere subjecti, and the other in genere causae formalis.
I wil prove that in your opinion you walk in a vitious circle, pro[ving] the self same by the word of God by the privat spirit, and the private spirit by the word of God.
The fourth thing that I am to show is to prove how you walk in a vitious circle, proving the selfe same by the selfe same, as the authoritie of the scripture by your private spirit, and your privat spirit by the authority of the scripture, by which manner of proof you may prove any thing.
Neither is here cōmitted any vitious circle between the authoritie of God & the church; as I have before convinced you in your grounds to commit. For first the authoritie of God revealing in vertue of which the infailibilitie of the proposition is beleeved, and the selfe same infallible proposition in vertue of which we beleeve that God [?]ies and reveales, hath two diverse objects. For the object of the infailible proposition is that God reveales; And the object that God reveales, or of the revelation of God is the veritie beleeves.
And now, (having done with your replies to the former matters,) I wil speak of those interlaced paragraphes which you bring in S. 98. &c: of the vicious circle as you call it, wherin you think we walk, proving (as you say) the authoritie of the scripture by the private spirit, and our private spirit by the authoritie of the scripture &c. But your Catholik opinion you say you will defend from such an idle proof and circular resolution of your faith.
Throughout this series of dueling arguments, the Ainsworths agree in defining a "vicious circle" (evidently on the authority of Aristotle) as a form of pseudo-logic that uses its parts to prove each other. The vice in a "vicious circle" thus lies in its inadequacy and deceptiveness as a claim to offer real proof of a proposition and, secondarily, in its entrapment of the deluded person's mind, which renders him incapable of seeing the falsity of the argument.
A number of other seventeenth-century religious polemicists use "vicious circle" in a similar way. In all, searches for the terms "vicious circle" and "vitious circle" at Early English Books Online yield some three dozen matches, all of them using it in the rhetorical sense of a logical argument that purports to prove its conclusion by citing an assumption whose validity depends on the validity of the conclusion.
A century after the Ainsworths, Charles Leslie, The Case Stated Between the Church of Rome and the Church of England in a Second Conversation Betwixt a Roman Catholick Lord and a Gentleman of the Church of England (1721) provides a helpful definition of the term "vicious circle":
G[entleman of the Church of England]. But what then is a vicious Circle ?
L[ord of Roman Catholic faith]. A vicious Circle is, when two Propositions are made Use of to infer one another without having any other Proof to support them. But if they be proved from other strong and convincing Reasons, this opens the Circle, and hinders it from being what we call a vicious one.
'Vicious circle' as a recurring series of unfortunate events
The earliest instance that I've been able to find of "vicious circle" in the sense of "undesirable or evil recurring chain of events" occurs much later, in "The Provincial Contest," in the Madison [Indiana] Republican (July 19, 1817):
If the submission is general, there reverts the great question of independent commerce. The [s]ystem of exclusion will be established simultaneously with the restoration of the authority of Spain. She knows no other mode of government. But this system of exclusion, the duration of which produced the first revolt, will not have become more tolerable nor more gracious in the eyes of the [Spanish colonial] Americans : it will of course, produce new insurrections : it is a vicious circle from which there is no escape. Is commerce free, is it independent, is it exclusive, or will it become an article of purchase? But in the re-establi[s]hment of the exclusive system, there will be found a new inconvenience ; that which will accommodate itself altogether to her commercial system ; those who are willing to adhere to their fidelity to Spain, will not wish to be ruined by their fidelity.
And from "Letter from Sydney," in the Sydney [New South Wales] Gazette And New South Wales Advertiser (March 18, 1830):
In the case of every plantation in North America, whether English, French, or Dutch, the settlers had to contend with a soil, which, being covered with dense forests, required much labour, and with a climate which incapacitated them for labour. It followed, of course, that they should struggle for a long while with hunger produced by sickness, and with sickness, again introduced by hunger. In every case many years, and in some cases more than 40 years elapsed, before the migrants escaped of that vicious circle. Now, in all that has been written of the untimbered grassy regions, and the glorious climate of Australia, there is not a word of exaggeration.
The phrase "vicious cycle" doesn't appear at all in the Early English Books Online database. The earliest occurrence of it that I am aware of is from Isaac Newton, Observation upon the Prophecies of Daniel, and the Apocalypse of St John (1733), page 246:
I know that EPIPHANIUS tells us, if some interpret his words rightly, that the JEWS used a vicious cycle [to compute their calendar], and thereby anticipated the legal new moons by two days. But this surely he spake not as a witness, for he neither understood ASTRONOMY nor RABBINICAL learning, but as arguing from his erroneous hypothesis about the time of the passion. For the JEWS did not anticipate, but postpone their months : they thought it lawful to begin their months a day later than the first appearance of the new moon, because the new moon continued for more days than one ; but not a day sooner, lest they should celebrate the new moon before there was any. And the JEWS still keep a tradition in their books, that the SANHEDRIM used diligently to define the new moons by sight : sending witnesses into mountainous places, and examining them about the moon's appearing, and translating the new moon from the day they had agreed on to the day before, as often as witnesses came from distant regions, who had seen it a day sooner than it was seen at JERUSALEM. Accordingly JOSEPHUS, one of the JEWISH Priests who had ministered in the temple, tells us that the Passover was kept on the 14th day of Nisan, κατα σεληνην, according to the moon, when the sun was in Aries. This is confirmed also by two instances, recorded by him, which totally overthrow the hypothesis of the JEWS using a vicious cycle. For that year in which JERUSALEM was taken and destroyed, he saith, the Passover was on the 14th day of the month XANTICUS, which according to JOSEPHUS is our APRIL ; and that five years before, it fell on the 8th day of the same month. Which two instances agree with the course of the moon.
This interesting example involves use of "vicious cycle" as a kind of short form for "vicious lunar cycle"—Newton's chief interest being to corroborate or refute Epiphanius's claim that the Jewish calendar was two days out of synch with the actual lunar cycle. As with the original sense of "vicious circle," the sense of vicious" here seems to be "fundamentally flawed" or "ill-reasoned," not "wicked" or "deplorable" or "reprehensible."
The next instance of "vicious cycle" that I found was much later, however. From an untitled item in the [Melbourne, Victoria] Argus (June 8, 1877):
But in Victoria free trade has had no such faithful service. One public man will palliate the fiscal offence, and will screen the offenders for party purposes. Another will pander to the cry for the sake of office. A third will deliberately sell his cause. Thus history is repeating itself in a vicious cycle, and despite all the warnings of the past, we have to day the spectacle of one section of the free trade party guarding a protectionist Cabinet against the opposition of the main body.
Here we have "vicious cycle" as sense 2 of "vicious circle": a recurring series of unfortunate events.
Usage commentators on 'vicious circle' and 'vicious cycle'
Here is an Ngram chart tracking the relative frequency of "vicious circle" (blue line) and "vicious cycle" (red line) over the period 1750–2005:
The most striking thing about this chart is how close the frequencies of the two phrases has become during the period from 1979 to 2005. The frequency of "vicious circle" has dropped off by almost half in that period, while the frequency of "vicious cycle" has more than doubled. As a result, "vicious circle" was only slightly more frequent than "vicious cycle" in works published in 2005.
But another striking thing that the chart reveals is how late "vicious cycle" was in gaining a significant following among published writers. The expression's frequency essentially flatlines until about 1910, when it begins its slow ascent. Because it is rather late to the party of everyday English "vicious cycle" doesn't show up in the works of early commentators who focused on correcting what they considered errors in usage.
Even as late as Bergen Evans & Cornelia Evans, A Dictionary of Contemporary American Usage (1957), a two-paragraph entry for "vicious circle" doesn't mention "vicious cycle" at all:
vicious circle was originally a term in logic, and still is. Vicious in this context means impaired or spoiled by a fault. Circle means a mode of reasoning wherein a proposition is used to establish a conclusion and then this same conclusion used to prove the proposition. It is called a circle because it has no real starting place—and, one might add, because there's no end to this sort of thing.. It is one of the most popular of all fallacies, the darling of the pompously ignorant.
The term vicious circle is commonly used to describe a situation in which solution of one problem creates other problems whose solution is incompatible with the original circumstances. Or, more loosely, some bad situation that, by its nature, seems to get worse and worse. In any use but as a term in logic the expression is now a cliché.
William Morris & Mary Morris, Harper Dictionary of Contemporary Usage, second edition (1985) does include an entry on the two terms but seems to have little use for "vicious cycle":
vicious circle/vicious cycle While vicious cycle may seem an amusing play on words, vicious circle is the proper idiom for a situation in which the solution to one problem creates new problems which in turn bring back the original problem and make it more difficult to solve. Thus the matter goes full circle, returning to its beginning. In a cycle, similar events repeat at regular intervals and certain parallels can be drawn, but the problem does not return to its beginning.
But eighteen years later, Bryan Garner, Garner's Modern American Usage, second edition (2003) accepts "vicious cycle" as legitimate alternative to "vicious cycle":
vicious circle; vicious cycle. Both mean "a situation in which the solution to one problem gives rise to a second problem, but the solution to the second problem brings back the first problem. Vicious circle is about 40% more common than vicious cycle in modern print sources.
And vicious circle is the phrase with the stronger precedent to support it. The OED records it from 1792 in the sense of "a situation in which an action and reaction intensify each other." Vicious cycle isn't recorded in the OED.
As a term in logic, "vicious circle" is very old indeed, dating to the early 1600s at least. Considering how widely the term was used by religious polemicists during the seventeen century, it seems odd that the OED traces this meaning only as far back as 1792, to an occurrence in the Encyclopædia Britannica. Here, at greater length, is that occurrence, from the entry for "Lightning" in the 1798 edition of the Encyclopædia Britannica:
In his explanation of the accident which happened to [James] Lauder, his lordship [Lord Mahon, later Earl Stanhope] is reduced to the greatest difficulty, and makes one of the most unphilosophical shifts in the world ; no less than that of arra[n]ging the clouds of heaven, not according to fact, but according to his own imagination. He supposes the existence of two clouds, one below the other ; and ascribes to them various motions and situations, which we have already taken notice of : but who knows whether such clouds ever existed? His lordship does not pretend that any body ever saw them ; and thus he runs into what is termed by logicians a vicious circle; he first assumes data, purposely made to accord with his hypothesis, and then proves the hypothesis from the data.
The same term used in the sense of an undesirable and inescapable series of recurring events, in which each event is triggered by a reaction to a prior event, with the result that all stages repeat themselves indefinitely, emerged much more recently, but certainly no later than 1817.
As for "vicious cycle," Isaac Newton used that term to describe supposed calculation of a "vicious lunar cycle" by ancient Jewish astronomers; but the term may not have appeared in its modern sense, as an alternative to "vicious circle" in contexts unrelated to logic, until perhaps the second half of the nineteenth century (user66974's answer points to a New York Times instance of relevant usage from 1858, which is nineteen years earlier than the earliest such instance that I was able to find).
My impression is that "vicious cycle" didn't appeal to English writers until people stopped thinking of the primary sense of "vicious circle" as being "a logical error involving a self-referential quasi-proof." Once the sense of "vicious circle" became more widely associated with "a recurring sequence of bad events," the variant "vicious cycle" gained a following because such a recurring sequence seems at least as cyclical as it is circular.