They are adverbs as answers to questions according to the Cambridge Grammar of the English Language p.570, Merriam-Webster and the American Heritage Dictionary. This seems pretty unanimous and seeing as the apparent lack of any subject or predicate can be explained by the following, there's no reason to cast them as verb-like.
The role they play in clause structure is that of a polarity adjunct, CaGEL p847:
Yes and no answers
Yes and no serve as markers of positive and negative polarity in
answers to questions. They may stand alone, or combine with a clause
that expresses the answer more explicitly:
7ia. A: Is this car yours? B: Yes (it is). / No (it isn't)
7ib. A: Isn't this car yours? B: Yes (it is). / No (it isn't)
7iia. A: He has gone, hasn't he? B: Yes (he has). / No (he hasn't).
7iib. A: He hasn't gone has he? B: Yes (he has). / No (he hasn't).
The choice between yes and no depends simply on the polarity of the
answer – not, for example, on agreement vs disagreement with what may
be suggested by the question. Polar questions, especially negative
ones, may be biased, indicating the questioner’s predisposition to
think that one or other answer is the right one, but that has no
bearing on the choice between yes and no. In [ib], for example, the
appropriate response is yes if the car is B’s and no if it isn’t,
irrespective of what A appears to expect is the case.
Similarly with answers to tag questions, as in [ii]. ∗Yes it isn’t
and ∗No it is are thus ungrammatical as single clauses. In Yes it is
and No it isn’t, the yes and no can be regarded as a special type of
adjunct, a polarity adjunct, which agrees in polarity with the clause
– a further case of polarity concord in English. The adjunct can also
be placed at the end of the clause, with prosodic detachment: It is,
yes and It isn’t, no. A response to [7i] with the form No, it’s Kim’s
would not of course violate the polarity concord rule, because here we
have not a single clause but a sequence of two, just as we do in No
it’s not mine, it’s Kim’s.
One respect in which the agreement vs disagreement factor is relevant
concerns the choice between single-word and expanded responses.
Suppose you ask Didn’t you post the letter after all, then?,
indicating that you think I didn’t. If in fact I did post it, I would
normally say Yes I did, not just Yes.
In short, the rest of the answer has simply been elided because it can be retrieved from context leaving behind the polarity adjunct only.
This is the same logic as:
A: When did you have breakfast? B: Early this morning.
where B's answer consists only of an adjunct of temporal location.
