The reason why a pair of is used for an item of clothing worn on legs is partly related to the history of clothing and tailoring. For example, pants (short of pantaloons, U.S. English) or trousers became a single garment late in the history. Before modern tailoring, such garments were made in two pieces, two seperate sleeves of fabric called a hose which were tied to a belt with braces and the open crotch was covered with breeches. The plural usage of these garments persisted out of habit even after they had started being made in one piece in modern tailoring. A shirt and similar upper-body garments were always a single piece of clothing, so they were singular.
Here is a diagrammatic reference of how trousers were made in the history before modern tailoring, from the article "Oldest known trousers found in China" of thehistoryblog.com:
Etymologically, the usage of a pair of is connected to how pair was borrowed from Anglo-Norman French (where the etymon is paire) around 1300; and ultimately from Latin paria. French uses the preposition de as seen in an Anglo-Norman example "deux peir de plates" from 1354, listed in the Middle English Compendium. In modern English, pair is restricted to meanings referring to 'two' or 'sets'. It is usually followed by plurale tantum nouns like pants, jeans, trousers or morphologically singular nouns that often come in sets of two like eyes, shoes, gloves. Historically, pair could be used without a preposition, sometimes with an unmarked plural; and it could also mean 'a few'. With the preposition of, pair was first used by Chaucer as "a paire of legges and of feet" (a pair of legs and of feet) in c1395 per OED. Chaucer's usage may have influenced the language also, as he has been called the father of English literature; and the semantic extension of the usage a pair of for garments worn on legs may be an analogy of its usage with legs.
Here are the definition of the original sense and usage of pair and the earliest citations from OED:
I. A couple; a set of two.
1.
a. A set of two individual things of the same kind, that are associated or complementary in use, purpose, position, etc.
Used esp. with reference to things worn or adapted to the right and left limbs or sides of the body (also colloquial with reference to the parts of the body themselves), and to other things used side by side or disposed symmetrically, as curtains, folding doors, etc.
to show a clean (also fair) pair of heels: see show v. Phrases 1a. to show a red pair of cheeks: see show v. 12b.(a) In singular, preceding the noun complement without of. Now only in abbreviated style.
c1300 St. Thomas Becket (Laud) 20 in C. Horstmann Early S.-Eng. Legendary (1887) 107 (MED) Euere he hadde ane peire feteres faste him up-on.(b) In singular and plural without complement or with of.
c1395 G. Chaucer Wife of Bath's Tale 597 He hadde a paire Of legges and of feet so clene and faire.(c) With unmarked plural (with or †without of). Now regional and nonstandard.
1432 Bailiff's Acct., Grantchester in Middle Eng. Dict. at Clouting For clowtyng of ij peyre schon.
Here are the definition of the relevant sense and usage of pair in question and the earliest citations of this sense from OED:
3. A single tool, instrument, or item of clothing, consisting of two joined or corresponding parts not used separately. Usually with of and plural noun complement, as pair of scissors, pair of trousers, etc.
a. In singular, preceding the noun complement without of. Now only in abbreviated style.
1391 in L. T. Smith Exped. Prussia & Holy Land Earl Derby (1894) 91 (MED) Pro furracione, j pair pynsons.b. In singular and plural without complement or with of.
▸ a1438 Bk. Margery Kempe (1940) i. 90 (MED) On was a maner of sownde, as it had ben a peyr of belwys blowyng in hir ere.c. With unmarked plural (with or †without of).
1543 in J. Raine Wills & Inventories Archdeaconry Richmond (1853) 43 Iij payre of pynsowrs, vid...ij cawkers, ijd.