In principle, this is a question of style, so different style manuals may give different recommendations. In practice, it seems that most, if not all, styles would say that if this signifies faltering speech or incomplete thoughts, then there should be no capitalization.
The Chicago Manual of Style says the following:
13.41: Faltering speech or incomplete thoughts
An ellipsis may be used to suggest faltering or fragmented speech accompanied by confusion or insecurity. In the examples below, note the relative positions of the ellipses and other punctuation. (For the use of ellipses to indicate editorial omissions, see 13.50–58.)
“I . . . I . . . that is, we . . . yes, we have made an awful blunder!”
“The ship . . . oh my God! . . . it’s sinking!” cried Henrietta.
“But . . . but . . . ,” said Tom.
Interruptions or abrupt changes in thought are usually indicated by em dashes.
3.55: Ellipses at the ends of deliberately incomplete sentences
An ellipsis alone (i.e., three dots with no additional period) is used at the end of a quoted sentence that is deliberately left grammatically incomplete.
Everyone knows that the Declaration of Independence begins with the sentence “When, in the course of human events . . .” But how many people can recite more than the first few lines of the document?
Have you had a chance to look at the example beginning “The spirit of our American radicalism…”?
Note that no space intervenes between a final ellipsis point and a closing quotation mark.
Examples from published literature
Searching through published literature on google books, we find that this rule is obeyed, with rare exceptions. Examples include
He… he opened her cloak, and he… touched her… (source)
How… how could he leave us? Now! When we need him most! He… he said he'd always be here! (source)
What… what, incredulous? (source)
What… what did you see? (source)
Some sources are inconsistent, like this.
Some cases where you would capitalize
Some cases where you would capitalize might be of interest.
Jill was probably finishing up her afternoon appointments. And he... He had another hour of paperwork before he could even think about leaving. (source)
This is not an instance of faltering speech/narrative. Rather, the words And he… are an intentionally incomplete sentence. This is done for a special effect. In speech, the pause here would be longer than the pause between your two whats, emphasizing that the protagonist needed to collect his thoughts and start a new sentence.
Tom paced the floor, walking back and forth all day long, repeating the word what. “What … What … What … What …,” nonstop, all day long, repetitiously. Of course, asking Tom “what” he was doing would only increase his repetitions. You see, Tom had found the secret of the universe in the word what. Tom believed he was Jesus Christ and that he was spreading the gospel truth through his words.
Tom, like many other people who are given the label schizophrenic, is extremely terrified of others.
Here, each what is meant to be a separate sentence, so in principle we could have had What. What. What. What. However, the ellipses are used instead of periods to indicate a longer than usual pause between these one-word sentences, and possibly a trailing intonation in prosody.