It's not very common, but people do it (and it is grammatically correct). I think it's a little more common in spoken English than in written.
(although I just did it in the sentence above)
As you noted, your example is shorthand for saying "He's careful, but not very careful". In that sentence, "very" is a modifier to "careful". If you leave off "careful", most English speakers will know what you mean by context.
As a side note, saying "He's careful, but not very" would make me think you were being casual/making a small joke. I think it's because the sentence ends in an adverb. If you were shooting for a serious tone, I'd say something more like "He isn't careful enough".
As for whether to use an ellipsis, I assume you mean:
- "He's careful, but not very."
- "He's careful, but not very..."
Both are valid, but they sort of have different meanings. The second one is less sure of yourself, and also more serious sounding (to me, at least). An ellipsis is how you write "this sentence kind of fades out, maybe somebody else can finish it."