The OP asked about the role of an infinitive following an adjective and followed by an object. I believe we can discern the syntactic nature of this construction by examining something similar called the tough construction, the syntax of which is discussed by Joasha Boutault in "A Tough Nut to Crack: A Semantico-Syntactic Analysis of Tough-Constructions in Contemporary English". Absolutely all this post following this initial paragraph down to the dividing line of tildes is a discussion of that paper, which is to say that absolutely none of that part is original to me, except for the examples, which I invented in an effort to follow the paper. Thus errors in the examples are mine and not Boutault's. The part after the tildes is mine and therefore I own any of its erroneous conclusions.
Tough constructions have the following variants:
Sticking with sentences like , and following Boutault, we can define the tough construction as a matrix clause with a main clause and a subordinate clause, the former consisting of
A subject, either a noun phrase () or a clause unto itself ([7a]), naturally in the nominative case and having a thematic role. I'm no expert in generative grammar, but I think this means the subject is the element most affected by the force of the verb
A stative verb
An adjective, optionally modified, that licenses a following clause
The latter consisting of
an infinitive clause with a transitive verb, either direct ([7a]), i.e, taking a direct object, or indirect ([7b]), i.e., taking a prepositional phrase as a complement.
the subject of which may be present ([8a]) or absent ([8b]) and
the object of which may be present only if the subject of the matrix clause is a dummy pronoun ([9a],[9b]).
 He is easy to get along with.
[7a] What the state has done is difficult to understand.
[7b] The topic is more difficult to think about in a crisis.
[8a] This is hard for me to hear.
[8b] This is hard to hear.
[9a] It is hard for me to hear this.
*[9b] This is hard for me to hear this.
From a generative grammar point of view, how do we generalize the cases with an object for the infinitive () and those missing an explicit object ([8b])? Since the missing object seems to be the same as the subject of the matrix clause, one solution is say that the missing object is raised. In , the car is picked, but if we have
[1b] You were wrong to pick.
say, in choosing up sides for a ball game, this means that both the wrongness belongs to you and you were the one picked. This apparently violates a cardinal rule of generative grammar, namely that co-referents cannot have different cases. Good thing, I suppose, since it seems to me that [1b] is ambiguous: it could mean you were the wrong one to be picked or you could be wrong to have made the choice.
Chomsky apparently solves this problem by declaring that the Adj-infinitive forms a single "complex adjective" through the application of various operators. For me, this is like talking about the distinctions between red and green unicorns.
Before examining a solution that avoids the pitfalls of these proposals, we take a detour into an interesting ambiguous sentence. Suppose you were at a ballet recital and asked an audience member, "Which one is your daughter?" whereupon you got the reply
 She's the beautiful dancer.
The adjective beautiful could have a narrow scope, applying only to the subject and thereby meaning, "No matter my daughter's talent for the dance, she's pretty and the rest are homely." On the other hand, beautiful could have a broader scope, applying to the subject and the action she effects, meaning, "She (and only she) is dancing beautifully, no matter how pretty she is." We can apply this dichotomy to the tough construction, finding two classes, which Boutault calls easytough and prettytough, named for the following canonical examples:
[11a] She is easy to get along with.
[11b] It is easy to get along with her.
*[11c] She is easy.
[12a] She is pretty to look at.
*[12b] It is pretty to look at her.
[12c] She is pretty.
Easytough constructions like [11a] allow the transposition to a sentence with a dummy pronoun subject like [11b], but do not allow the deletion of the infinitive clause like [11c]. ([11c] is grammatical but has an entirely different meaning from [11a].) It's the opposite for prettytough constructions like [12a]: there's no grammatical transposition to the dummy pronoun ([12b]), but the infinitive may be deleted without changing the sense ([12c]).
This might indicate that the adjectives easy and pretty modify different things. In prettytough constructions, the adjective gives a defining property of the subject, in a manner analogous to the narrow interpretation of . This is why the infinitive may be dropped: the subject is still there. And it's why the dummy subject won't work: the adjective is trapped between a dummy pronoun and an infinitive, neither of which are appropriate for the defining adjective to modify.
However, in easytough constructions, the adjective modifies the subject only through the activity or state associated with the infinitive, as in the broader interpretation of . In this case, the dummy transposition works because the infinitive is still there. Dropping the infinitive won't work because the infinitive is necessary to provide the state or action.
Dropping the infinitive in prettytough constructions keeps both grammaticality and the sense of the original. Not necessarily the entire sense, though. This is hard to see with pretty since what other way can she be pretty except by looks? But consider
[12aa] The car is cheap to buy.
[12ca] The car is cheap.
In both cases, it's the car that's not expensive, but [12ca] allows for the possibility that the car is cheap to operate. These considerations signal that in prettytough constructions, the infinitive is an adjunct.
In easytough constructions, the infinitive cannot be dropped, making this obligatory construction a complement to the adjective.
The operators of generative grammar that Boutault proposes to make the adjunct/complement distinction work is far beyond my ability to comprehend. As far as I'll go is note Boutault's claim that almost all tough constructions have an evaluative meaning and not a purely descriptive one. Thus we do not say the descriptive
*[13a] it was dark to film
We have to say something like
[13b] it was too dark to film
allowing the adverb to provide the evaluation. I gather that the operators necessary to generate the proper parse trees work on the scope of the evaluation. (Don't take my word for it, because I think we're now talking about a herd of unicorns.)
Now all of this generative machinery is in place to take care of the (generally) absent object of the infinitive, the exception being the object's appearance in the case of a dummy pronoun subject of the main clause ([9a]). In all cases, the object, explicit or implicit, is a co-referent of the subject. But in the OP's example, we have
[14a] You were wrong to pick that car.
In this case, the object of the infinitive (that car) is different from the the subject, which refers to the errant person addressed. However, it's now the subject of the infinitive that's co-referent with the subject of the main clause, i.e., the errant party and the picker are the same. And it's the subject of the infinitive that is generally absent:
*[14b] You were wrong for you to pick that car.
appearing only when a dummy pronoun appears as the subject of the main clause ([14c]).
But notice we can still apply the rules to see that [14a] is analogous to the easytough category. We may transpose to a dummy pronoun version
[14c] It was wrong of you to pick that car.
and we can't drop the infinitive without changing the sense:
***[14d] You were wrong.
Which leads to the conclusion that the infinitive clause is a complement to the adjective.