So I'm wondering if being terse or succinct is a connotation of pithy, or is it not necessary. Can a long sentence or exposition that has substance be called pithy?
The OED's entries include (emphasis mine):
2a. Of language or style: full of concentrated meaning; conveying meaning forcibly through brevity of expression; concise, succinct; condensed in style; pointed, terse, aphoristic.
Source: Oxford English Dictionary
Where all the highlighted terms convey a sense of not using more words than is necessary. So, while a phrase/sentence described as pithy will often be a relatively short, I do not believe it has to be. A longer sentence could still be described as pithy, provided it is about as short as it can be while still getting its meaning across. Contrast with wordy, verbose, long-winded etc.
Etymologically speaking, the "terse, succinct" meaning would seem to come (at least indirectly) from the "fruit" related meaning. For pith the OED has as a "concrete use":
I. Concrete uses.
1. The soft internal tissue of a plant part; esp. a central cylinder of parenchyma in a stem or root. Also: a layer of spongy tissue lining the rind in certain fruits, especially citrus fruits, in which it is white and often bitter-tasting.
with dated citations back to c1150 (and an earlier undated use). For its "abstract" use (as pertains to sentences), it has both:
II. Abstract uses.
4.a. The innermost or central part of a thing; the essential or vital part; the spirit or essence; the core, the nub. Frequently in pith and marrow.
5.b. Force, power, energy (of words, speech, etc.). In later use chiefly: the quality of conveying meaning forcibly through brevity of expression; succinctness, conciseness.
First emphasis theirs; second mine
with dated uses back to around 1425 for sense 4.a. and 1531 for sense 5.b..
There would seem to be a progression from pith meaning the inner/internal tissue (of a plant), through the innermost or central part (the core or essence) before getting to "core" meaning of a phrase or sentence, by dint of it being brief, concise or succinct.