This is far from being a complete answer to an interesting question, but in ‘An Invitation of Old English and Anglo-Saxon England’, Bruce Mitchell provides some evidence that the past participle was seen as an adjective in the ancestor of modern perfect constructions. There were three forms in Old English that correspond, for example, to our past perfect construction. One is: Hie hæfdon ϸone cyning ofslægenne, ‘They had the king, slain.’
Ofslægenne is inflected as an adjective here, showing the -ne ending of the accusative masculine singular. It agrees with ϸone cyning, the object of hæfdon.
Similarly, the ancestor of today’s passive is seen in patterns like
He wæs / wearϸ ofslægen ‘He was / became slain’
Hie wæron / wurdon ofslægene ‘They were / became slain’. In the second, the participle is again inflected as an adjective.
As for the term itself, I suspect we use ‘past participle’ simply because most of our traditional grammar terms are based on descriptions of Latin grammar. Latin had present active and future active participles and perfect passive and future passive participles. The perfect passive participle has come to be understood as a past participle when applied to English.