If you say that 'to/modal + have + past participle' is a single construction, you need to explain what makes it different from 'have + past participle'. In other words, you need to explain why the constructions 1. 'to
[1] to + havehave + past participle'participle
and 2. 'modal
[2] modal + havehave + past participle'participle
are more closely linked with each other than either is with 3. 'nothing
[3] (nothing + have) have + past participle'participle.
Yes, you can say, "well'well, 'to/modal +[1] and [2] have + past participle' has an extra word as compared to 'nothing + have + past participle'with [3]'." But what matters for grammar is not the word count, but rather how a thing behaves grammatically, that is, what kinds grammatical functions it can fulfill and howfulfill—how it interacts with other grammatical objects. And in those respects, as should become obvious from the discussion below, there is nothing that links 'to + have + past participle'[1] and 'modal + have + past participle'[2] more closely than either is linked with 'nothing + have + past participle'[3].
In fact, if anything, it is 'to + have + past participle'[1] that is the 'odd man out' here. As far as grammatical functions, 'modal + have + past participle'[2] and 'nothing + have + past participle'[3] are far more similar to each other than either is to 'to + have + past participle'[1]. Namely, they both involve a finite verb, whereas the 'to' construction[1] involves a non-finite verb. As we shall see, this is a very important difference as far as grammatical behavior.