I've provided definitions for "pen" as a verb (which YOU should have provided! That's OK, you'll know for next time).
If you're asking whether the verb "to pen" can include using a pencil, then the answer is clearly yes, as shown by the dictionary definitions below.
To write or compose: penned a letter.
(tr) to write or compose
Collins English Dictionary
Write or compose.
Oxford Living Dictionaries
write, indite * pen a letter
14. to write with or as with a pen; put down in writing: to pen an essay.
15. to draw with or as with a pen: to pen a sketch.
7. to write or draw with or as if with a pen: to pen an essay.
Random House Kernerman Webster's College Dictionary
So the only definitions here that explicitly mention using a pen in the verb definition of "pen" are the last two. And they don't require a pen, they're written in terms of "with or as if with a pen" or "with or as with a pen".
Merriam-Webster defines the verb as to "write". And I don't think it's controversial that you can write your name using a pencil.
I personally wouldn't use your phrasing of "pen your signature". Have you ever gone to fill out a form before stopping yourself and wondering "I wonder if both pen and pencil are acceptable." The verb "pen" to specifically mean using an actual pen most likely IS ambiguous. Take the following examples:
"Previously unpublished letters penned by struggle hero, Nelson
Mandela, primarily during his 18-year imprisonment..."
Can we tell whether he used a pencil/pen/crayon/chalk/smartphone stylus/finger painting? OK, so some of those are crazy, but pencil or pen are both likely possibilities.
The song was penned in George’s family home after an average Sunday
dinner and became an iconic Christmas song that made the playlist we
all listen to every Christmas on a loop.
This is in reference to a song. Assuming these were lyrics and not music notes, do we know whether a pen was used? No.
If you hand someone a form and a pen and say "pen your name" or pen something else, then it's more obvious you want them to use a pen, more specifically the pen you gave them. But it's better to avoid this phrase in my opinion. "To pen" is often used in place of the more ordinary "write" maybe in an attempt to sound more creative because "write" is seen as common and prosaic. Also "To pen your signature", I'm not sure about that. Wouldn't you rather be more clear and say something like "Please sign your name" or "Please write/provide your signature."?
The alternative in your question:
"Please sign your name legibly and in pen."
Leaving apart the fact that, as jlovegren pointed out, signatures are often illegible (intentionally or unintentionally), the last part of the request specifically states your desire that they write their name/signature in pen.
So yes, it is very often ambiguous.