You're on the right track when you use the term implied, because the subject of the sentence is the implied you, even though the word you is not in the sentence. This answer, by the way, is from a non-grammarian.
The sentence as it is is in the imperative mode, which we use to give commands or make requests. You could expand the sentence and not change its basic meaning by saying:
"[You] give [to] me the pencil."
The OBJECT is the pencil. The implied SUBJECT is the implied (i.e., missing) you. The person who gives the pencil to the requester is the subject, since in an imperative sentence, the person who is "subjected to" the command can say yes or no. As we sometimes say, "The ball is in his court."
If, as a parent of a preschooler, I see my child playing with a sharpened pencil, I will say to him, "Give me the pencil!" (Actually, I would more than likely snatch it from his hand.) The child could decide no, and keep the pencil. We could say he is disobeying an "order" from his boss by refusing to be the subject of my imperative.
That, however, is part of the risk of issuing commands: the one to whom the command is given can say yes, no, or "later, dude."
That's how it goes when two people have their eye on the same object, which is in the possession of one of them. The person who holds the pencil in his hand is really the subject of the command. If he says no, a fight might ensue. If he says yes and hands it over, case is closed (for now). Of course he could ruin the pencil by breaking it in half, but that scenario is for another day.