Square is a hyponym of rectangle, which is a hypernym of square. The wikipedia article Hyponymy and hypernymy says
In linguistics, a hyponym is a word or phrase whose semantic field is included within that of another word, its hypernym (sometimes spelled hyperonym outside of the natural language processing community). In simpler terms, a hyponym shares a type-of relationship with its hypernym. For example, "pigeon", "crow", "eagle" and "seagull" are all hyponyms of "bird" (their hypernym); which, in turn, is a hyponym of "animal".
Edit: For more precision, one should add qualifying phrases like “the word” or “the set”:
- the word square is a hyponym of the word rectangle
- the set of squares is a subset of the set of rectangles
However, I think subclass [a term mentioned in Joshua Taylor's comments] can be used without added qualifiers.Wiktionary gives the following definition of subclass relevant within computing: “In object-oriented programming, an object class derived from another class (its superclass) from which it inherits a base set of properties and methods”.
- squares are a subclass of rectangles
Speaking mathematically, a class is a less-determinate category or collection of things than is a set. From en.wiktionary, class means “A group, collection, category or set sharing characteristics or attributes”, and subclass means “A rank directly below class”.