The earliest use of the term, 'muscle-car', in print, that I could find dates back to 1964...and it refers to a Land Rover (British-made):
No concessions are made to style in this muscle-car; frills are few. Nobody tried to prettify the Rover. For example, there's not a significant speck of chrome on or in it. Belt-line molding and bumper are heavily galvanized instead. Even the grille is heavy wire mesh--galvanized.
(From Popular Mechanics, Jan 1964.)
The Land Rover, of course, does not match what the term came to mean.
The next appearance of 'muscle car' in print, in Popular Science (Oct 1965), does refer to a small car (for the time) with a large-displacement engine--typically a V8:
DODGE CORONET becomes "muscle car" with Hemi-426 engine, but relies on drum brakes only. Rally suspension makes car remarkably well-balanced and good-handling.
As is suggested by the "scare quotes", the term 'muscle car' was recently adopted with the sense of 'small car, large displacement engine', American-made at the time, but about that more below.
The OED Online doesn't hop on for the ride until the term's meaning is established in general journalistic use, beyond specialty magazines. Then, the use is attested with a quote from the Mar. 20, 1966 Chicago Tribune Mag.:
Gran Turismo Omologato. Our muscle car with all the would-be tigers in its wake.
["muscle, n.". OED Online. December 2015. Oxford University Press. http://www.oed.com/view/Entry/124020?redirectedFrom=muscle+car (accessed January 21, 2016).]
The OED Online presents a more restrictive definition than might meet general approval now and historically:
muscle car n. chiefly N. Amer. a high-performance car; spec. one of a class of sports cars having V-8 engines, produced in the United States mainly in the 1960s and 1970s.
Historically, 'muscle cars' have been made elsewhere than in the USA. Australia made some in the 1960s, 1970s and 1980s. Holden (part of General Motors at that time), Ford Australia and Chrysler Australia all contributed models, although they were handicapped early in the game by racing rules that required 200 of the cars had to be sold to the general public, along with a government ban of 'supercars' after some high profile incidents:
The first instance was a Wheels magazine journalist driving at 150 mph (240 km/h) in a 1971 Ford XY Falcon GTHO Phase III 351 cu in (5.8 L). While the car was getting exposure in the press, the second incident occurred in George Street, Sydney, when a young male was caught driving at an estimated 150 mph (240 km/h) through the busy street in a 1971 Ford Falcon GTHO Phase III, drag racing a Holden Monaro GTS 350. This was known in Australia as "The Supercar scare".
(From "Muscle car", at Wikipedia.)
Generally speaking, 'muscle car' refers not only to a 'small car with a large-displacement engine', usually a V8. It also designates a particular appearance and characteristics in contrast to European automobiles, as described by Peter Henshaw (and paraphrased at Wikipedia):
"The Muscle Car is Charles Atlas kicking sand in the face of the 98 horsepower weakling." Henshaw further asserts that the muscle car was designed for straight-line speed, and did not have the "sophisticated chassis", "engineering integrity", or "lithe appearance" of European high-performance cars.
(Internal quotes from Muscle Cars, P. Henshaw, 2004. Paraphrase from Wikipedia article cited previously.)
However, if Italy, Germany, Japan or whatever country designs and builds a car that meets 'muscle car' enthusiasts' approval, there's no doubt that car also will be called, and billed for sale, as a 'muscle car'. Where the car is made has little to no importance to many of the people that buy and drive 'em.
Regarding why muscle cars are not called 'supercars' and 'sports cars', the reason why not, as might be expected, is going to come down to an unverifiable matter of opinion. However, my semi-educated guesses are as follows:
- 'Sports car' is and was in general use for a broader category of cars.
- 'Supercar' is and was, like 'sports car', a more generic term in general use, without the specificity of 'muscle car'.
'Supercar', for example, is attested as early as 1927, with reference to an ideal car (that is, not one that existed):
European manufacturers believe that the automobile of the future will readily travel 90 or 100 miles an hour, and that superhighways will be built to allow a tremendous speed. The car of the future must be a supercar.
(From Boys' Life, Jun 1927.)
The next attestation from the Google Books corpus suggests that, by the time the 'muscle cars' were a product looking for a term, 'supercar' had been pre-empted and diluted not only by earlier use, but by company branding:
Frank Hrubetz Company, ride manufacturer, announces the appointment of two agents for the distribution of Spit Fires in Europe. The Supercar Company, Belgium, will handle sales in Europe except the Scandinavian ...
(From Billboard, May 24, 1947.)
Attestations in the aforementioned corpus then proliferate through the 1940s and 1950s, which to me suggests that the American car manufacturers making muscle cars, and the enthusiasts buying them, might well have wanted to settle on any term except 'supercar'--that term had been diluted and branded with a meaning that did not respect the unique qualities of the 'muscle car'.
This comment from an EL&U contributor helps clarify and explain the distinction between the supercar and the muscle car:
"American Muscle Cars" were designed around straight line drag racing while "European Super Cars" were designed around Le Mans and FIA racing. "Drag Racing" vs "Le Mans" is what drove the division. – K. Alan Bates
'Sports car' had a similar history, without the specific branding; it was used to designate a broad range of small, sporty, but not necessarily straightforwardly 'muscular' cars. Sports cars were known for speed and tight handling on twisty roads, while 'muscle cars', when they appeared on the scene, were not as suited to fast driving on curvy roads. Note that the muscle car may be considered a subclass of sports car by some enthusiasts and observers.