Whence, I often asked myself, did the principle of life proceed? It was a bold question, and one which has ever been considered as a mystery; yet with how many things are we upon the brink of becoming acquainted, if cowardice or carelessness did not restrain our enquiries.
I think the conditional sentence here is grammatically correct as Shelley wrote it. It is an example of an open conditional, in which the main clause (apodosis) has an present tense (indicative), and the if-clause (protasis) has a past tense (indicative) verb, indicating something prior to the protasis. The past tense is not a past subjunctive. This is the case despite the author - and possibly Frankenstein - implying that the if-clause is false, or doubtful, a case where a remote ("subjunctive") conditional is typically used. The open conditional is used for rhetorical effect. as part of a rhetorical question, marked by the inversion "are we". Its implication that its speaker does not know truth of the protasis or apodosis is to Frankenstein's benefit if he is to be considered courageous rather than rash in having tackled the "bold question".
To come to this conclusion, I found it very helpful to read the chapter 8 section 14 of the Cambridge Grammar of the English language (CGEL), which covers conditionals. The "rules" often used for teaching English, as described on say Wikipedia or Grammarly, and more accurately by Cambridge, classify conditionals roughly as below.
- zero (if + present, present - something generally true)
- first (if + present, future - something likely in present / future)
- second (if + past, would etc - something unlikely or impossible in present / future)
- third (if + past perfect, would have etc - something impossible in past),
- mixed (a mixture of second and third)
While these are broadly descriptive of common patterns, they are inadequate and in some cases misleading or wrong if a deeper understanding is required, as in this case.
The Cambridge Grammar classifies conditional sentences instead as Open, which roughly corresponds to zero and first conditional, or "indicative" conditionals in traditional grammar, and Remote, which correspond to second and third conditionals, or "subjunctive" conditionals. They have three important properties in common (14.1.1):
(i) Invariant meaning: [...] P & Q are related [...] to exclude the combination the P is true and Q false.
(ii) Consequence implicature: P is a consequence of Q
(iii) only-if implicature if not-P, then not-Q
Open conditionals are distinguished by the speaker not knowing (or not wishing to imply) whether the protasis and apodosis are true or false (CGEL ch8 14.1.1). Remote conditionals are characterised instead by the implication that the protasis is false, or, better, by consideration of the consequences in a different world where the protasis is true (14.2.1). Importantly, both Open and Remote can have protases and apodoses relating to any combination of past, present or future time-frames (14.1.2 and 14.2.2).
To support my conclusion, we have to refute some notions implicit in most versions of "the rules", and also in the original question.
Firstly, Open conditionals can use past tenses, including a past (indicative) tense in the protasis combined with a present in the apodosis. The tense pattern in Shelley's sentence is therefore compatible with it being an Open conditional, despite "the rules" often not mentioning this possibility. It indicates a temporal relation: the protasis is in the past relative to the present of the apodosis. Obviously Open conditions are more likely to relate to the future because we generally know less about it, but there are also times when we don't know something about the past - or don't want to admit to knowing something about it! An example:
I regret it if I offered you a cup of tea
[apodosis present, protasis past]
This is a conditional expression of remorse for something that the speaker does not necessarily want to accept happened in the past, but does not deny either. Compare with the Remote options "I would regret it if I offered you a cup of tea", which a very different meaning, implying that you will not be offered one (in the present or future); and "I would regret it if I had offered you a cup of tea", which implies that I didn't in fact offer you one in the past, and that I'm glad that I didn't - or that my lack of regret now should be taken as evidence that I did not in fact offer the tea in the first place! (If all this seems like a lot of fuss over a cup of tea, try imagining it's the morning after the night before, and that the tea might be euphemistic).
Another example comes also from Frankenstein, and uses the same tense pattern as the sentence in question:
"I believe I am; but if it be all true, if indeed I did not dream, I am sorry that I am still alive to feel this misery and horror." (Frankenstein ch21)
[protasis "did not dream" past, apodosis "I am" present]
Here Frankenstein is I think considering what would be a painful truth, as shown by "if it all be true" (the present subjunctive "be" is now archaic, but this is not relevant to the question at hand). The protasis "I indeed I did not dream" shows him considering a past about which he apparently has imperfect knowledge (was he dreaming?) and drawing a real, present, conclusion from it ("I am sorry"). He could have said "if indeed I did not dream, I would be sorry", but this diminishes the reality both of the regret and of his acceptance that he may not have dreamt - unless it were followed with "and indeed I do feel sorry", in which case it would show the reasoning bringing him to the painful conclusion that the real world does resemble the Remote but seemingly-not-so-counterfactual one. The alternative "mixed second / third conditional", "if indeed I had not dreamt, I would be sorry" would suggest he thinks he did dream (unless we as readers knew that he did not in fact dream, in which case it's denial).
Secondly, we should note that Remote conditionals cannot have a simple present tense in the apodosis, but rather need a (typically preterite) modal, most commonly "would" (14.2.2). But we have a simple present here, so either Shelley has made a grammatical mistake - hence the original question - or we have an Open conditional. Since her other writing does not show similar errors, I would favour the latter explanation in the absence of conflicting evidence.
The Cambridge Grammar does give an example of a Remote condition which appears to have a present tense in its apodosis:
"If you needed help, Mary is willing to help"
[protasis past, apodosis apparently present tense] (14.2.2 example 51)
It explains this as containing an implied "you would be interested to know that Mary is willing to help". This may seem slightly contrived, but in any case I don't think it is a comparable example, because there is no causative relation between the need of help and the willingness: presumably, Mary is willing to help even if you don't need the help. In Shelley's sentence, a such causative relation is clearly implied, and is necessary to the argument.
Thirdly, we should note the past tense in the protasis ("did not restrain") is not explicitly marked as past subjunctive (or modally remote preterite, in the Cambridge Grammar's more helpful terminology), and so we cannot assume it to be one. In English, "to be" is the only verb which shows a difference between past subjunctive and indicative (in traditional grammatical terms), and it does only in the 1st and 3rd persons singular: "I/he/she/it were" is subjunctive, whereas "I/he/she/it was" is indicative. Thus, only if the protasis were something like "if cowardice was/were not a restraint on our enquiries" - or if she had written in a language like Italian or Latin that shows the difference in all verbs - would Shelley have had to make a decision on verb forms which could help clarify the matter.
There are two pieces of evidence could support the notion the apodosis verb is an implicit subjunctive (modally remote preterite). The first is that the "rules" generally associate a past tense with a Remote ("second" or "subjunctive") conditional. We have already seen is not valid, because Open conditions can also use past tenses.
The second is that Shelley and Frankenstein do seem to be implying that the protasis is false: that cowardice and carelessness DO restrain our enquiries, and this is what Remote ("second" or "third") conditional are generally used for. This is more complex to account for. While it is true that Remote conditionals generally imply that the speaker thinks the protasis is false (in the real world), it is also possible to have Open conditionals where the protasis is considered to be unlikely or impossible by the speaker. This is often for done rhetorical effect. Shakespeare has this example:
FREDERICK: Thus do all traitors;
If their purgation did consist in words, [protasis - past tense]
They are as innocent as grace itself. [apodosis - present tense]
Let it suffice thee that I trust thee not.
(As You Like It, Act 1 Scene iii)
Here Frederick clearly does not think that speaking eloquently can rid traitors of their guilt. We have the same tense pattern as Shelley's example, and we can similarly substitute "would be" for "are" in the apodosis to obtain a similar meaning. But note that the Open condition (along with the indirection of "their" and "they" rather than "thy" and "thou [art]") allows Frederick to suggest that he knows that he doesn't know for sure whether Rosalind is a traitor: instead he limits himself to saying he doesn't trust her, perhaps despite her (real world) grace.
(Legal reasoning is an obvious use case for rhetorical Open conditionals relating to the past in modern English: a lawyer we might want to establish the truth of a conditional relation before concerning himself with its protosis, even though he has a view on the truth of the latter: e.g. establishing the relation "If somebody committed the murder, he wore that glove" might come before showing that the glove does not fit a defendent.)
The Cambridge grammar offers another example. Its explanation is worth quoting in full:
Note also that there are occasions where a remote conditional would inappropriate even though I know that P [the protasis] is actually false:
 a. If Grannie is here she is invisible
b. If Grannie were here she'd be invisible
The context here is one where someone has suggested that Grannie is here and I wish to pour scorn on the idea. This is achieved by the open conditional [a], where I show that that suggestion has an absurd consequence: since Q ("Grannie is invisible") is patently false, P ("She is here") must be false too. The rhetorical effect is drastically diminished in the remote version. Here I imagine a would differing from the actual one in that P is true. But once we envisage a world that differs in one respect from the actual one, the possibility arises the it could differ in other respects too: perhaps in this imaginary world people can be invisible. One can still argue from [b] that Grannie is not actually here [...], but the demonstration is less immediate, less direct than in the open version. Similarly with  above: "If that's Princess Anne, I'm a Dutchman" is a conventional way of ridiculing the idea that that is Princess Anne, but we do not say "If that were Princess Anne I would be a Dutchman".
(CGEL Ch8 14.2.1 example 37)
In other words, an Open conditional can be a better choice than a Remote conditional as a rhetorical device for immediacy and directness, even when its protasis is considered unlikely or false.
Shelley's underlying argument is more complex than the example above, but I think there are significant parallels. I think it goes something like this:
- If cowardice etc. does not restrain us (P), we get close to making discoveries (Q) (open condition, judgement of truth of P and Q suspended, we merely establish that they are related)
- Progress has not yet been made in the real world ("has ever remained a mystery") (not Q)
- Therefore cowardice has restrained us in the past in the real world (not P) (this is similar to the scorn in the example above)
- But I am not cowardly ("it was a bold question") and I have worked hard ("I often asked myself") in the real world
- Therefore, cowardice did not restrain me ("us"? - I perhaps a royal "we" / "we" of modesty) in the real world (P)
- Therefore, I (we) have got close to making discoveries in the real world (Q) (conveniently, it's only close, so I don't have to actually say what they are, but maybe I'll explain to you if you listen....)
- Implicitly, my working on such bold questions is justified, because I've done something good, getting us closer to new knowledge (Q).
In other words, Frankenstein is imploring us to think of his boldness as a virtue which others have lacked, rather than as a futile attempt to answer impossible questions. The open condition makes things more vivid, allowing implications to be drawn about the real world and not some other one that might be different, as with the "Grannie" example.
To understand the effect, it might also be useful to consider the sentence with the (exclamative) rhetorical question removed:
yet we are upon the brink of becoming acquainted with many things, if cowardice or carelessness did not restrain our enquiries.
And also a possible simplification of the sentence in more modern language:
Where does life come from? It was a bold question, and one that has always seemed beyond us. Yet how close are we to great discoveries, if cowardice and carelessness did not hinder our research?
These are perhaps not the formulations a modern-day speaker is most likely to produce, but despite their relative convolution, they still work: they are grammatical, and the underlying rhetorical ideas are still relevant in modern language.
There are two possible Remote equivalents of Shelley's sentence. The first ("second conditional") lacks the temporal relation between the protasis and apodosis:
yet with how many things would we be upon the brink of becoming acquainted, if cowardice or carelessness did not restrain our enquiries
It loses the implication that work done in the past creates progress in the present. Instead it suggests that we need to be not cowardly in the present (or future, or timelessly) to make progress in the present (or future). Using it therefore risks losing the implications that Frankenstein has already done progress-making work and / or that he has been courageous and painstaking.
The second is the most logically equivalent Remote form of Shelley's sentence, with the temporal relation preserved because the apodosis verb in the past perfect, marking both pastness and Remoteness:
yet with how many things would we be upon the brink of becoming acquainted [today], if cowardice or carelessness had not restrained our enquiries [in past]
This carries the implication that cowardice or carelessness have in fact restrained our enquiries in the past: both other people's and Frankenstein's! This is not helpful if he wants to justify his tackling a "bold question". (The ambiguity of the first person plural is a very useful rhetorical device for Shelley and Frankenstein: it can represent "you and me", "everybody", "people in general", "A modest me", "Me and people similar to me" or "a generous me, you can share in my good work....". This is worth comparing with "they" in the example from Shakespeare).
Thus, for Shelley's purposes, an open conditional is the best device, because it allows us to suspend judgement on whether "cowardice and careless restrain our enquiries", while still allowing a broader implication that it is true. A rhetorical question is used to draw out significant implications on both sides. Shelley's brilliant mind is guilty not of bad grammar but rather of a rhetorical or poetic concision beyond the dreams of this explanation!