I've found references that this term is originally used for the purchase of stocks that not listed on the stock exchange.
Adjective phrase over-the-counter is attested from 1875, originally of stocks and shares.
It seems like it could mean two things:
- Stock is sold retail rather than through an exchange. The exchange takes place simply by handing the stock "over the counter" rather than through a third party exchange.
- Stock is sold "on the other side of the counter" rather than "behind" the counter which implies unlisted.
I would be appreciative if someone with an OED subscription wouldn't mind taking a look and seeing if maybe this was derived from another term or if some of the earlier documented references of its use could give some hints.