The aristocratic usage theory
In a comment (now vanished, along with several others that once appeared beneath the original question), a commenter opined that the different ways of handling the possessive of a proper name reflected the diction appropriate to a member of the aristocracy ("Lord Nonsuch his") and the diction appropriate to members of the lower classes ("Bibber's").
The argument based on class distinctions is hampered by the fact that the people who use the form "Lord Nonsuch his" in the play's script are, as the original question notes, not members of the upper classes; rather, they are a sponger (Failer), the wild gallant himself (Loveby), and some bailiffs. So the only plausible way to argue that Dryden's shift between "Lord Nonsuch his" and "Bibber's" (for example) is class-based is to claim that all of the play’s characters, regardless of their own class, follow the form "Lord Nonsuch his" because they recognize that form of possessive as attaching to aristocratic or otherwise illustrious persons and to no one else.
But elsewhere in the play, Mrs. [Frances] Bibber (the wife of a tailor) uses an apostrophe-s possessive form in connection with the Queen, which punches a large hole in the high-born distinction theory:
Frances [Bibber]. I will speak, that I will, so I will : What! Shall I be a Dresser to the Queen's Majesty, and no Body must know on't?
If possessives of persons of high rank were systematically accorded possessive pronoun treatment instead of apostrophe-s treatment, we would expect Mrs. Bibber to refer to her prospective position as "a Dresser to the Queen her Majesty"—but she doesn't.
The consonant ending theory
In another comment (also unfortunately obliterated), StoneyB offered the insightful idea that Dryden's differential handling of "Lord Nonsuch his" and "Bibber's" might have been based not on any special form reserved for aristocratic possessors, but on differences in final consonant or final consonant sound in each word. StoneyB also suggested that Dryden might have adopted the (soon to be abandoned) scholarly thesis that all instances of "X's," where X is a person's name, were actually contracted forms of "X his." (I hope that I have recalled and represented his comment accurately.)
With regard to the consonant ending theory, you might wonder about the significance of the fact that "Bibber's" is a two-syllable word, just as "Bibber" is, versus the fact that "Nonsuch" is a two-syllable word, but "Nonsuch's" adds a third syllable to the original word. One result of that extra syllable is to make "Nonsuch's" much closer in sound to "Nonsuch his" than "Bibber's" is to "Bibber his." So if Dryden subscribed to the notion that (for example) "Bibber's" was a contraction of "Bibber his," he might have been inclined to render "Nonsuch" as "Nonsuch his" on grounds that the latter form was "more proper" and yet would not sound odd to an audience hearing the words spoken in a play.
But what are we to make of Dryden's handling of the possessive of the last name "Trice"? Characters in The Wild Gallant refer to the possessive form of that name on five separate occasions—always as Trice's. One example:
Burr: What Noise is that? I think I hear your Cousin Trice's Voice.
And another:
Setstone. Having drawn him [Loveby] aside, I told him, if he expected Happiness, he must meet me in a blind Alley I nam'd to him, on the back-side of Mr. Trice's House, just at the close of Evening ; there he should be satisfied from whom he had his supplies of Mony.
Here, "Trice" has one syllable and "Trice's" two, so that basis for the distinction between "X's" and "X his" seems not to be sustained. But the sound of the final consonant of the proper name is not dispositive, either. Consider this example:
Failer. One way there is. Sir Timorous, pray walk a turn while Burr and I confer a little upon this Matter——Look you Burr, there is but one Remedy in Nature, I vow to gad : That is, for you to have a new Sir Timorous, exceeding this person in Bounty to you. Observe then, in Sir Timorous his place will I go, and i'gad I'll marry my Lady Constance ; and then from the Bowels of Friendship bless thee with a thousand pounds, besides Lodging and Diet for thy Life.
The final s sound is indistinguishable in "Trice" and "Timorous," but Dryden consistently uses "Trice's" and (on the only occasion when it comes up) "Timorous his."
Evidently Dryden bases his styling of the possessive not merely on the sound of the final voiced consonant, but on the presence or absence of subsequent unvoiced letters in the name's spelling (such as the silent e in "Trice").
In this respect, Dryden seems generally to have applied his rule consistently—at least in his plays of the middle 1660s. For example, in An Evening’s Love (1668), Dryden has this:
Alonzo. I know what you would say, Sir, that though I am your neighbor, this is the first time I have been here——[To Bellamy] But come, Sir, by Don Lopez his Permission let us return to our Nativity.
And in The Maiden Queen (1667), the final lines of the play are these:
Queen. The Pow'rs above, that see/ The innocent Love I bear to Philocles,/ Have giv'n its due Reward ; for by this means/ The Right of Lysimantes will devolve/ Upon Candiope ; and I shall have/ This great Content, to think, when I am dead,/ My crown may fall on Philocles his Head.
But in a play written twelve years later, Dryden appears to have changed his mind. In Oedipus (originally published in 1679, though I'm quoting an edition published in 1735) you can find at least six instances of Laius', plus one Tiresias’ and one Adrastus'. For example:
Oedipus. I am satisfy'd./ Then 'tis an Infant-lye ; but one Day old./ The Oracle takes place before the Priest ;/ The Blood of Laius was to murder Laius :/ I'm not of Laius' Blood.
Here, Dryden doesn't add an s after the apostrophe, but he (or his editor) does use an apostrophe rather than introducing the discrete word his to indicate the possessive, as he had done with the possessive of Philocles in The Maiden Queen.
'X's' or 'X his' as applied to possessive forms of women's names
I don't know whether Dryden's handling of possessive forms of proper names was purely idiosyncratic or whether a certain stratum of literary men of his time generally upheld it, but—influential though Dryden was—it seems to have had little or no staying power. One problem with his approach that I haven't noted previously involves how to handle possessives associated with women's proper names. If Dryden believed that the apostrophe-s possessive was simply a shortening of "his," how could he justify using it in the following instance?
Loveby. Here Mr Bibber, pray put Madam Bibber's Name into the Warrant.
Shouldn't that say "Madam Bibber her" or maybe "Madam Bibber'r"? Even worse is this instance:
Frances [Bibber]. Nothing vexes me, but that this flirting Gentlewoman should go before me ; but I'll to the Heralds Office, and see whether the Queen's Majesty's Dresser should not take place of any Knight's Wife in Christendom.
You can hardly argue for "The Queen his Majesty his Dresser"—but that would appear to follow from Dryden's insistence on "Lord Nonsuch his daughter" and "Philocles his Head."
Cautionary note on editions of Dryden’s work
After encountering the split in possessives between "Lord Nonsuch his" and "Bibber's" in a 1735 edition of Dryden's works, I looked for as early an edition of The Wild Gallant as I could find in Google Books search results. The earliest version I could find in a first round of searches was from 1718—55 years after the play's debut. That is the edition I read when searching for other instances of Dryden's handling of possessives in the play.
More recently, however, I came across an edition of The Wild Gallant from 1701; and though the spelling and punctuation are for the most part the same as in the later edition, one noteworthy difference involves the double possessive spoken by Frances Bibber, which the 1701 edition renders as "the Queen's Majesties Dresser." As a matter of simple coherence, "the Queen’s Majesty's Dresser" (the 1718 version) seems better, but Dryden may well have used "Majesties" (and "Queens") in 1663. Likewise, one of the five instances of "Trice's" in the 1725 edition is rendered as "Trices" in the 1701 edition, although the omission of the apostrophe here by the 1701 editor appears to be accidental.
Nevertheless, it strikes me as very possible that in the earliest editions of the play there may have been no apostrophe-s possessives at all. Snippet views from editions of the play from the 1960s (which may have reintroduced Dryden’s original punctuation) include lines such as this one:
Frances. Nothing vexes me, but that this flirting Gentlewoman should go before me; but I'll to the Heralds Office, and see whether the Queens Majesties Dresser should not take place of any Knights Wife in Christendom.
Intriguingly, by 1718, "Majesties" had become "Majesty's" and "Queens" and "Knights" had acquired apostrophes, but "Heralds" still had not. Meanwhile "Nonsuch his" is firmly in place in all of these editions. (In some editions issued in the 1800s "Nonsuch his" became "Nonsuch's.")
If in his original script for The Wild Gallant Dryden avoided using apostrophes for possessives such as "Bibbers," his desire to avoid "Nonsuchs" is perhaps more understandable. But the use of "his" under one set of conditions and the omission of it under others remains a consistency problem; the gender issue of the s in "Madam Bibbers Name" is not resolved; and indeed the respelling of "Majesty" in its possessive form as "Majesties" introduce another layer of special rules to handle particular proper name endings ("spell the possessive form of a proper noun ending in y as though you were spelling the plural of the proper noun"). You could argue that generating the possessive form "Bibbers" from "Bibber" follows the same pseudo-pluralizing path, but a simpler way to express what actually goes on in that case is with the rule "add s to the proper noun to create the possessive form"; moreover, formulating a rule for possessives on the basis of plural forms doesn’t make sense if you take the position (as Dryden seems to do) that the possessive form is actually based on the proper name plus his.
Conclusion
Dryden's handling of the possessive form of proper names ending in ch, s, z, and so on during the 1660s may have been internally consistent, but the rules that he adopted seem arbitrary and unintuitive. Also, to the extent that Dryden intended his approach to promote a notion of the possessive apostrophe-s (or the possessive closed-up s) as emerging from "his," it runs into a serious problem with the possessive form of women’s proper names. Ultimately his entire project on behalf of "X's" (or "Xs" without the apostrophe) in some places and "X his" in others seems insupportable under any reasonable theory that I can imagine.
UPDATE (March 28, 2017)
I just came across an interesting discussion of the origin of this practice in Joseph Priestley, The Rudiments of English Grammar, third edition (1772):
The apostrophe denotes the omission of an {i} which was formerly inserted, and made an addition of a syllable to the word.——Mr. Pope, and some of his contemporaries, to avoid the harshness in the pronunciation of some genitives, wrote the word {his} at the end of the word ; as Statius his Thebais, Socrates his fetters (Spect[ator]) imagining the {'s} to be a contraction for that pronoun : But analogy easily overturns that supposition ; for Venus his beauty, or Men his wit, were absurd.
An uncredited repetition (with considerable amplification) of this wording appears in an anonymous review of Richard Bentley, A Dissertation upon the Epistles of Phalaris," in the Critical Review (January 1777):
We find the e [before s in old spellings of the genitive singular] frequently retained by some of our ancient writers. Thus, in the verses on Seint Vonefrede, which, according to bishop Fleetwood, are near five hundred years old, or perhaps much older, the author writes, kinges sone, and Goddes grace. Gower, who lived in the fourteenth century, says Goddes folke, Goddes sande {a Saxon word signifying mission or being sent} worldes welth, mannes helth. Chaucer, who wrote about the same time, has Goddes sonne, Christes sake, worldes transmutacion, kynges lawe, ladyes name, knyghtes tale, mannes voice, childes play, Agenores doughter, Philippes sonne, Cupides bowe, &c. {Edit. 1542.}
Our old English writers were however extremely inaccurate in the termination of the genitive case. The poets followed no rule in this respect; but sometimes inserted the e, and sometimes left it out; sometimes cut off, and sometimes added a syllable, for the sake of the measure.
Bishop Lowth observes, that 'God's grace was formerly written, Godis grace;' and Dr. Johnson remarks, 'that knitis is used for knight's, in Chaucer.' But this, we apprehend, is an irregular mode of spelling, not supported by analogy, or agreeable to the original formation of the genitive case.
Several eminent writers, to avoid a harshness in the pronunciation of some genitives, have subjoined to the substantive the pronoun his: as, "Asa his heart," 1 Kings. xv. 14. "Christ his sake." Liturgy. "The first book of Statius his Thebais." Pope's transl. of Stat. "Socrates his fetters were struck off." Spect. No 183. "Ulysses his bow." Guard. No 98. Mr. Addison tells us 'that the s represents the his and her of our forefathers.' Spect. No 135. But analogy easily overturns this supposition : for 'the queen his palace,' 'the children his bread,' would be absurd.
We therefore conclude, that the termination of our genitive case in 's is regularly derived from the Saxon; and that the apostrophe implies the omission of the letter e, as we have already observed.
This discussion, which may have been written by Priestley and certainly borrows from the shorter commentary in his Rudiments of English Grammar, points to two possible motives for expressions of the form "Lord Nonsuch his": as a visual way of abating the oral harshness of "Lord Nonsuch's"; and as a fancied return to an ancient English form of expressing possessives as "[possessor] his."
The harshness avoidance theory is interesting, but it doesn't explain, for example, the title A Petite Pallace of Pettie his Pleasure from 1576, noted in my original question. But perhaps in that instance the justification for the form was to avoid an awkward diereses-style pronunciation along the lines of A Petite Pallace of Pettieës Pleasure. That rationale—extending the category of pronunciations to be avoided beyond harshness to include any awkward, difficult, or otherwise unpleasant result— might reconcile the omitted e theory of apostrophe-s possessives (which the anonymous reviewer favors) with the harshness avoidance theory (which he attributes to writers who adopted the "[possessor] his" usage) in this instance.
FURTHER UPDATE (January 30, 2021)
One point that I mentioned only glancingly in the section on editions of Dryden's work is that the occurrence of "his" as a variant for apostrophe-s as a possessive may have originated at a time before the apostrophe-s possessive came into vogue.
The predicament that not having an apostrophe-s option put writers (and printers) in is suggested by the title of a book written by Lancelot Andrewes sometime prior to 1626 (the year of his death) but published posthumously in 1657: Apospasmatia Sacra, or, A Collection of Posthumous and Orphan Lectures Delivered at St. Pauls and St. Giles His Church. Here, it seems to me, the writer is dealing with a situation where a person using modern punctuation conventions would have replicated what I take to be the then-current pronunciation of the two churches as "St. Paul's" and "St. Giles's." In the absence of an apostrophe-s option, the printer very naturally renders the first church's name as "St. Pauls." But how can the printer replicate "St. Giles's"? Under the circumstances, "St. Giles his" appears to be a more effective way of replicating the presumed pronunciation than "St. Giless." That rationale—or something like it—may well have influenced writers and printers to introduce "[Name] his" as a possessive form in cases where the final letter in the name made it problematic simply to add an unpunctuated s to the end of the name.
