The verb refine is "awkward" here. It could mean 1) purify, distil, make more concentrated (referring to something mentioned earlier) or 2) fine-tune, revise, make more precise (referring to the timing of the visits).
There's also ambiguity as to whether we're talking about times when it would be possible to visit (regardless of whether we do or not) or times when we actually manage to visit (by implication, with some difficulty). But I'll ignore that aspect.
The short answer to OP's question is that if there is a "something mentioned earlier" (sense #1 above), the word it is required, to refer back to whatever that "something" is. If not (sense #2), it should not be present.
But I think perhaps what's really confusing OP is better illustrated by looking at a similar construction involving when and an ambiguous / dummy it...
I like [it] when you (infinitive verb phrase)
I like it when you smile (10 hits in Google Books)
I like when you smile (5 hits)
...where both forms do occur, but they mean exactly the same thing. As the figures suggest, we usually do include that "dummy pronoun", but I think it would be pointlessly pedantic to argue that either version is more "correct". Consider this more "generic" example...
I fall asleep as soon as I go to bed at 11 o'clock every night, but I always know [it] when you come home after midnight
...where know could be replaced by various alternatives (hear, resent, hate, like, record1,...). And in all cases, the word it is optional - if present, it can be thought of as a "forward reference" to the activity specified later (your smiling, your coming home late), or simply a "dummy".
My feeling is this kind of optional "dummy it" can only occur with verbs of perception / reaction. It certainly can't apply with OP's verb refine, or with any "synonyms" for either of the two senses I defined in the first sentence of my answer.
1 I thought it was worth including something like record here even though it requires a bit of "creative context" to make sense (perhaps an unhappy wife recording her drunken husband's behaviour, to support her pending divorce application). The point being that recording the time of coming home is effectively a "reaction" to that event.