In
Winnie saw the truth at once, knowing as she did the character of her, whom, if she had ever looked upon as a mother, must from this moment forfeit every claim upon her feelings, unless it were that of utter contempt.
the parenthetical and other additions to the matrix sentence complicate analysis.
Stripping to an easier but comparable (grammatically at the salient point) stripped example:
Winnie met the woman who must acknowledge that she had no right to expect any sympathy from Winnie.
The parenthetical in the original does not control the grammar of the matrix sentence, so nominative 'who' is the correct pronoun.
(As FF points out in a comment, the parenthetical should be [and this even if Winnie had ever regarded her as a mother]. Parentheticals need to fit with the matrix sentence better than is the case in the original.) Also, 'unless it were' would nowadays sound better rendered 'except perhaps for a feeling of utter contempt'.