NOAD says:
matter (n.): physical substance in general, as distinct from mind and spirit; (in physics) that which occupies space and possesses rest mass, esp. as distinct from energy.
substance (n.): 1 a particular kind of matter with uniform properties : a steel tube coated with a waxy substance.
2 the real physical matter of which a person or thing consists and which has a tangible, solid presence.
It's easy to see why you'd have trouble with the two words. Their definitions are circularly defined, so there is some obvious synonymous overlap.
That said, there are certain contexts where one word sounds inherently more appropriate than the other. Here are a few examples:
Water and ice are the same substance
.
There was a stange yellow substance
left on the windshield.
No substance
in the world would give a human being the strength of Superman.
Much of the matter
reconverged after the supernova explosion.
Satellites stay in orbit due to the earth's gravitational pull on matter
.
Stars are made up of hydrogen, helium, and a small percentage of other matter
.
The most common substance
in stars is hydrogen.
In such sentences, how would someone know when to use matter, and when to use substance? As I pondered this question, the best technique I was able to devise came from keying on this part of NOAD's first definition of substance:
a particular kind of matter
I realized, if I could use "[a] particular kind of matter" in the sentence, then the word substance would probably be the better fit. For example:
Water and ice are the same matter.
Water and ice are the same particular kind of matter.
The second sentence conveys the sentiment more accurately; so, it would be better to use the word substance. Similarly:
There was a stange yellow matter left on the windshield.
There was a stange yellow particular kind of matter left on the windshield.
Again, substance is the better fit. However:
Much of the matter reconverged after the supernova explosion.
Much of a particular kind of matter reconverged after the supernova explosion.
In that case, the meaning is obscured by inserting "a particular kind of", so matter is the better word to use.
Satellites stay in orbit due to the earth's gravitational pull on matter.
Satellites stay in orbit due to the earth's gravitational pull on a particular kind of matter.
Once again, gravity affects all matter, not just a particular kind of matter used in satellites, so matter is the better word.
I'll repeat this exercise for another sentence from my initial examples:
Stars are made up of hydrogen, helium, and a small percentage of other matter.
is OK, because I would not say:
Stars are made up of hydrogen, helium, and a small percentage of another particular kind of matter.
because there are more than three elements in most stars. However, I could say:
Stars are made up of hydrogen, helium, and a small percentage of other particular kinds of matter.
meaning I could also say:
Stars are made up of hydrogen, helium, and a small percentage of other substances.
Lastly:
The most common substance in stars is hydrogen.
is left as an exercise for the reader.
I wouldn't go so far as to say this is a hard and fast rule for determining when to use matter or substance, but it doesn't appear to be a bad initial analysis.
One other footnote: this methodology works when the two words are being used in a similar context, and would not apply to other contexts, such as a financial matter, or the substance of an argument, or to idiomatic uses (such as gray matter for intelligence).