"The Old Lady who?" comes from a similar expression, fairly idiomatic, where a person, knocking on a door for example, states their first name, "Mary", and the person on the other side of the door asks for clarification by using this specific expression:
The first person would know to clarify,
"Mary Angeles". [their last name]
This exchange is idiomatic. You wouldn't hear someone say "Mary what?" or "Which Mary?" in the same idiomatic way, although you might surmise their meaning from either phrase.
Knock-knock jokes rely on an audience familiar with that exchange, which is the setup of the joke. You have to know to respond to "Old Lady" with the specific "Old Lady who?"
With that said, in the context of a knock-knock joke, you wouldn't necessarily ruin the joke if somebody responded "what". (Grammatically, this doesn't work. We prefer the pronoun "who" when clarifying something animate and "what" for clarifying something inanimate.)
The joke is broken into five steps. The punchline comes from the funny man's corruption of either his own response in step three or from the straight man's response in step 4. So, if the joke comes from the funny man's corruption of step 3:
- Straight Man: Knock knock.
- Funny Man: Who's there?
- Straight Man: Lettuce?
- Funny Man: Lettuce who?
- Straight Man: Lettuce in. It's freezing!
You could get away with swapping "who" and "what", though it wouldn't sound as good.
However, Your particular joke comes from the corruption of Step 4, and therefore requires the straight man to respond with the idiomatic "Old Lady who?" to provide the yodeling sound.