“The library, which I had never seen otherwise than wide open, was tight shut.”
otherwise = in the other way or differently
than = compared to something
otherwise than = differently or in the other way compared to something
You never saw the library differently or in the other way compared to "wide open".
"differently/in the other way" modifies "saw"
differently from "wide open" means in the way that not wide open = closed or = opened with short distance
You never saw the library closed/in the other way before.
In the given context, otherwise than means that the person has only seen the library open before, and is now seeing it closed.
Try replacing with 'other than'. They mean the same thing and are both grammatically correct, though 'otherwise than' embellishes the sentence a little more.