The 1792 usage appears to be earliest available in the scientific sense. The figurative usage established later, mid-19th century as shown by the Phrase Finder :
A vicious circle was the name given by 18th century logicians for a fallacious proof in this form:
- A depends on B - B depends on C - C depends on A
This was alluded to in Edition 3 of The Encyclopedia Britannica, in 1792:
- "He runs into what is termed by logicians a vicious circle."
A wider use of the expression was taken up by the medical profession and there are several examples from the early 19th century of it being used to describe conditions where one symptom affects another and the health of the patient steadily deteriorates.
The figurative, that is, not specifically logical or medical, meaning became established in the middle of the century; for example, this piece from Henry James' Notebooks, 1892:
- "The whole situation works in a kind of inevitable rotary way - in what would be called a vicious circle."
Examples of "vicious" in the sense of flowed reasoning can be found in the 17th century as suggested by The Grammophobia:
Logicians in the early 17th century used the term “vicious” (from the Latin vitiosus, meaning faulty or defective) to refer to a flawed syllogism.
Here’s an OED citation from 1697: “If from true premisses follows what is false, it is a sign that the form of the syllogism is vitious.”
By extension, the phrase “vicious circle” was used in the 1700s for an argument that circles back on itself because its premise is flawed (usually the premise is used to justify the conclusion, which in turn is used to justify the premise).