You are right, it is fine to use "on the other hand" without explicitly mentioning "the one hand" - the reader/listener can easily infer it.
Indeed, using "on the one hand... on the other hand" in most contexts sounds laboured and overwrought. It is sometimes useful to signpost to the audience in advance that you are going to supply an opposing view later, in which case it is useful - but for the most part it is better to leave the first part out.
In support of this: I spent a little time perusing the British National Corpus. It reports 5311 uses of "on the other hand", but only 1417 of "on the one hand". That would seem to suggest that "on the one hand" is only used roughly one third of the time, but in fact it's even less than that, because (judging from a random sample) most uses of "on the one hand" contrast it with "on the other" (or not at all) - so it's probably closer to one in four.