An example from Milton's Paradise Lost: "If thou beest he (...)". The similarity to the German word bist is something that I found really interesting. I was wondering when, how and why the word got pushed out of the language.
Definitely by the time "thou" had got pushed out of the language
As other people have said, the -(e)st conjugated verb forms like "beest" were generally only used in sentences that had the old second-person singular pronoun thou as the subject, so your question may reduce to the question of when, how and why thou fell out of use.
Fairly suddenly in the 17th century, thou began to decline in the standard language (that is, particularly in and around London), often regarded as impolite or ambiguous in terms of politeness. It persisted, sometimes in an altered form, particularly in regional dialects of England and Scotland farther from London, as well as in the language of such religious groups as the Society of Friends. [...]
In the 18th century, Samuel Johnson, in A Grammar of the English Tongue, wrote: "in the language of ceremony ... the second person plural is used for the second person singular", implying that thou was still in everyday familiar use for the second-person singular, while you could be used for the same grammatical person, but only for formal contexts. However, Samuel Johnson himself was born and raised not in the south of England, but in the West Midlands (specifically, Lichfield, Staffordshire), where the usage of thou persists until the present day, see below, so it is not surprising that he would consider it entirely ordinary and describe it as such. By contrast, for most speakers of southern British English, thou had already fallen out of everyday use, even in familiar speech, by sometime around 1650.
"Thou beest" vs. "Thou art" (in the indicative)
That said, before thou had fallen completely out of use, art was in competition with beest as the 2nd-person singular present tense indicative form, and it seems that in the most well-known dialects, and in the standard language, "thou art" was either winning or had largely won the competition.
Goold Brown's "Grammar of English Grammars" from 1882, quotes Robert Lowth's grammar of English, published in 1762, as saying that
"Be, in the singular number of this time and mode, especially in the third person, is obsolete; and is become somewhat antiquated in the plural."—Lowth's Gram., p. 36.
The connection of forms like "beest" with the "subjunctive" seems to be complicated and unstraightforward
The issue of defining when forms "be" and "beest" are "subjunctive" is actually fairly complicated. There have been differences in the use of the subjunctive mood across the course of English, but there also have been differences in the use of "be" forms as indicative vs. subjunctive forms. As described in Quuxplusone's answer to the question "Why “the powers that be”?", the word "be" could be used as an indicative form (especially in the plural) in the time of Shakespeare.
The OED says of forms like "beest":
Historically, these continue indicative forms, but apparently at least some early modern English texts prefer beest (rather than art) in syntactic environments that otherwise favour use of the subjunctive such as conditional clauses
As far as I can tell, the original 2nd-person singular subjunctive form of the verb be would have been be: Wiktionary's conjugation tables say that Old English used bēo for the singular of all persons in the present subjuctive, and that Middle English used be (of course, Wiktionary conjugation tables don't show the variety of forms that were actually used in a time period, and there are sometimes coding errors that cause them to have incorrect forms, but in general I find Wiktionary a useful reference for information about old languages).
In the past tense, the history of the forms wert and wast seems to be similarly complicated. The OED says
In Old English the 2nd singular past subjunctive of this verb (in common with all strong verbs) is not formally distinct from either the 2nd singular past indicative (as it would be for a weak verb) or from the 1st and 3rd singular past subjunctive; in its later history it continues to be problematic as a formal category. This section illustrates use of the 2nd singular in syntactic environments that typically trigger use of the past subjunctive; it is not entirely certain to what extent these should be regarded as subjunctive forms.
You cna see more complicated notes about "beest" and "wert" if you use Google Books to look at the linked page from Brown.
"in the language of ceremony ... the second person plural is used for the second person singular",...Samuel Johnson
Two ancient pronouns collapsed from the mid 16th Century to the mid 17th Century, thou and ye. All the research and speculation that might be done probably will not produce any single smoking gun that caused this.
English was probably following the example of other Western European languages by replacing the traditional 2nd person with a formal form of address for social superiors. That a 2nd person plural form came to be used was normal for the process. Ye gave way to you with the French vous often pointed to as the model. There is an issue with you having the same vowel (ou) as vous also the same as thou and the vowel being pronounced differently in you and thou; that might support the vous theory. But, prior to the collapse of thou and ye, honorifics such as "Your Grace" or Your Mercy" were used and you might have been thought of as a convenient shortening. You or something similar had been the 2nd person plural object form. English was not a uniform language 500 years ago, and isn't now.
"Thou beest" was the normal 2nd person singular subjunctive while the traditional 2nd person singular was still in general use. The Milton quote in the question might be rendered "if you were he" in current English, but might also be written "if you are he".
There is not so much a replacement of beest with are as a loss in general use of the 2nd person singular. While the 2nd person singular survives in some narrow special and local uses, it only exists in standard English today in some archaic phrases. You is now standard for both singular and plural, and is grammatically a plural form.