Early recorded use of 'negatory'
One of the earliest instances of negatory that a Google Books search finds is in Elisha Coles, An English Dictionary, Explaining the Difficult Terms that are used in Divinity, Husbandry, Physick, Philosophy, Law, Navigation, Mathematicks, and other Arts and Sciences (1717), which provides the following simple definition:
Negatory, belonging to Denial.
This entry follows two related terms:
Negation, a denying.
Negative, belonging thereto, also that Manner of Expression.
Clearly negatory and negative had very similar meanings at that time, though negative was (and remains) by far the more common term of the two.
Examples of negatory in the wild include this one in a legal decision, dated January 21, 1702, in The Decisions of the Lords of Council and Session, From June 6th, 1678, to July 30th, 1712 (1761):
David Grant wright against Daniel Simpson writer to the signet. Each of them having a tenement at the Netherbow, Daniel claimed a passage or entry through David's land to his own, and stopped a fyre [?] for carrying off the water : whereupon David raises a reduction, and a negatory action of declaratory of his immunity and freedom from any such servitudes ; and that the close is his own, and the little shop therein, and so cannot be made a common entry by Daniel, &c.
Another early instance of negatory used in a legal context occurs in William Strahan's 1722 translation of Jean Domat, The Civil Law in Its Natural Order, Supplement to Book IV Of the Publick Law, Title I, "Of the Several Sorts of Judicial Demands and Actions," VI:
When a Real Action is commenced by the Owner of a House or Lands, claiming a Service due from the House or Lands of another Person, that Action is called a Confessory Action. And the Name of Negatory Action is given to that which is brought by a Person, who insists that his House or Lands are not charged with the Service which is claimed to be due from them.
This definition seems well suited to the facts of the 1702 Grant v. Simpson dispute quoted above, where negatory appears.
We also have this nonlegal instance from "Bavius versus Prompter" in The Gentleman's Magazine (March 1736, quoted from The Grubstreet Journal, number 324):
[B]ut now he has chang'd the Dispute to Whether these Ifs are any Denials at all ; and to prove that they are not, quotes Philip. iv. 8. If there be any Virtue, and if, &c, adds it may be as well asserted that these Ifs are implicit Denials of what is contained in that Verse—But any one who compares the Prayer with this Place of Scripture may see the contrary : The Word If is there used in an affirmatory Sense, but in several other Places in a negatory one, particularly Psalm cxxx, 30.
And from Benjamin Martin, Bibliotheca Technologica; or, a Philological Library of Literary Arts and Sciences (1740):
TERMS are either Positive or Negative. Positive Words have an affirmative Sense, and signify some Positive Idea ; as Art, Life, Sense, Motion, &c. But negative Terms exhibit negative Ideas, or have a negatory Sense express'd by some Particle or Preposition of Denying join'd to them ; as Artless, Lifeless, Senseless, Nonsense, &c. ... But in many Cases Positive Words have Negative Significations, and the contrary, which is a great Imperfection and Unhappiness in Language.
This last example is particularly interesting because the author frequently uses the word negative in the same discussion and seems to introduce the term negatory to indicate a shade of difference, as if negative meant "negative" in the modern sense, while negatory meant something like "negating."
Johnson's Dictionary of the English Language (1756) omits negatory. Nevertheless, the term continues to appear (though never commonly) in serious works over the ensuing decades. For example, we have instances from law such as George Bowyer, Commentaries on the Modern Civil Law (1848):
Thus a negatory action contains an assertion; and the right of servitude in the defendant being disproved, the right becomes established as part of the dominion or ownership of the proprietor, the residue of which was not in dispute.
And from "Notice of Caldwell's Nicomachean Ethics," in The Classical Journal (December 1828):
This shorter form of the negatory adjective is (it should be observed) quite agreeable to analogy: thus we have λóγος, λóγιμος, αλογος, not αλóγιμος ; μóρος, μóριμος, αμóρος not αμóριμος, &c.
And from history in Thomas Carlyle, The French Revolution: A History, volume 3, The Guillotine (1837), writing about Caron de Beaumarchais, the author of Figaro:
"At midnight" (it was but the 12th of August yet), "the servant in his shirt," with wide-staring eyes, enters your room: — Monsieur, rise; all the people are come to seek you; they are knocking, like to break-in the door! "And they were in fact knocking in a terrible manner (d'une façon terrible). I fling on my coat, forgetting even the waistcoat, nothing on my feet but slippers; and say to him" — And he, alas, answers mere negatory incoherences, panic interjections.
And from logic in Augustus De Morgan, Formal Logic: or, The Calculus of Inference, Necessary and Probable (1847), who describes a class of "negatory conclusions" that he defines in contradistinction to "affirmatory conclusions." He later remarks:
Our conclusion is that no negatory complex syllogism is of any more logical effect than the strengthened particular derived from it. Thus we may say that, so far as the extent and character of the inference is concerned, the former is the latter.
Between 1850 and about 1975, many occurrences of negatory in Google Books search results involve law or logic (or both), though authors specializing in other areas such as history, psychology, and philosophy use it, too.
"Negatory" after the coming of 'Convoy'
The year of change for negatory was 1975, when the citizen's band radio fad blossomed in the United States, and C.W. McCall released his hugely popular recording "Convoy," which uses (among other bits of contemporaneous CB radio slang) the word negatory in the sense of "no":
Ah, you wanna give me a 10-9 on that, Pig Pen? Negatory, Pig Pen; you're still too close. Yeah, them hogs is startin' to close up my sinuses. Mercy sakes, you better back off another ten [miles].
If some people now view negatory as being strictly or primarily a joke word, CB radio slang may well be to blame. The humor (I suspect) derives from the perceived intellectual overreach of typical users of the term, since CB radio enthusiasts as a group are not widely viewed as being especially well educated. A similar instance of overreach is reported in the U.S. Supreme Court case, Brandenburg v. Ohio (1969), where a Ku Klux Klan spokesperson is recorded on film giving a speech that, among other things, uses the novel terms revengent and revengeance:
The Klan has more members in the State of Ohio than does any other organization. We're not a revengent organization, but if our President, our Congress, our Supreme Court, continues to suppress the white, Caucasian race, it's possible that there might have to be some revengeance taken. [395 U.S. 446]
As I recall, the Klansman's neologisms were a source of considerable amusement in the Constitutional Law class where I heard the case discussed.
Whatever the general perception of negatory may be, serious writers continue to use it completely seriously, as in Nicholas Tavuchis, Mea Culpa: A Sociology of Apology and Reconciliation (1991):
The ultimate negatory stance in a culture that does accord a place for apology would be the categorical denial of any obligation to respond to the call or, for that matter, even to give an account for an offense.
And Steven M. Cerutti, The Word of the Day: The Unlikely Evolution of Common English (2005):
Ancient Greek has only one negatory prefix, the single letter alpha (α).
As for how negatory and negative differ in non-CB-radio-derived usage, negatory clearly works in far fewer contexts than negative does, because it has far fewer meanings—essentially, "denying or negating"—whereas negative applies as well to such areas as electromagnetic polarity and a cynical or pessimistic attitude. Still, within the narrower confines of "denying or negating," the terms are very similar. Just be aware that some readers or listeners are likely to reflexively think "Ten-four, good buddy" whenever they encounter the word negatory.
I should also point out that, despite its regular (though infrequent) use in certain areas of English for more than 300 years, negatory is not accorded an entry in Merriam-Webster's Eleventh Collegiate Dictionary (2003) or its predecessors, suggesting that the people at Merriam-Webster have never regarded it as an everyday English word in good standing.