A Concise Etymological Dictionary of the English Language (Walter W. Skeat, 1910) says grounds comes not from having being ground, but from being left over at the bottom.
grounds, dregs. (E.) So called from being at the bottom. Cf. Gael. grunndas, lees, from grunnd, bottom., ground; Irish gruntas, dregs, from grunnt, the bottom.
An 1825 patent talks of:
an improved apparatus for the purpose of beneficially separating the infusion of tea or coffee from the grounds or dregs.
A French and English dictionary (Randle Cotgrave, 1673) translates:
Fondrilles: The grounds, lees; or dregs of liquor.
There are definitions of sediment and faeces that give grounds as a synonym.
The justice of the peace, and parish officer, Volume 2 (Richard Burn, 1814):
Penalty on brewer receiving stale beer grounds, or mixing with any liquor except malt and hops, 100l.
Beer and ale grounds were not uncommon in recipes, laws for unscrupulous landlords, instructions for making files, as an ingredient for sloughing ulcer treatments, medical comparisons with vomit.
But nowadays, it is chiefly applied to coffee: