I don't really know if there are any strict rules governing the use of "On" in titles. As GoDucks said, I think it can largely be regarded as a custom or tradition that isn't as popular as it once was.
The best example that I can think of is Darwin's On the Origin of Species, which is a scientific work, rather than a philosophical treatise.
As far as I know, the word "On" isn't commonly used in scientific works any more, but I think the word commonly appears in popular titles. It would be helpful if I could think of some examples, but I'm kind of stumped at the moment, other than On Golden Pond - however, I think "On" has a different connotation in that particular title.
Here are a couple relatively modern titles: On the Meaning of Life, On the Meaning of Sex.