Both previous answers have pretended (implicitly) that the text read
XHTML […] subsets HTML.
But in fact, the sentence reads (again, only the relevant fragment):
… document types that … subset HTML.
It’s those document types that, each on its own, are subsets of HTML. This explains the apparent contradiction that XHTML both extends and subsets HTML: parts of XHTML extend HTML, whereas other parts are subsets of HTML. (As an example of an XHTML document type that extends HTML, consider XForms, and for a subset of HTML, consider XHTML 1.0 strict.)
Either way, the usage of “subset” as a verb is meant to mean “be a subset of”.