Most dictionaries define a prefix as
a letter or group of letters added to the beginning of a word to make a new word. (Cambridge)
So if you follow that, you can identify a prefix by removing it and seeing if you are left with a word or rather a part of a word that has no meaning in English.
EDIT: Prompted by a certain sigh, I am adding a more accurate definition of prefix:
an affix attached to the beginning of a word, base, or phrase and serving to produce a derivative word or an inflectional form (M-W)
In his article Some Important Prefix Types, Paul Fanning follows the same line:
post- in postpone and posthumous is not a prefix because -pone and -humous are not possible words. (guinlist).
But if you dig in the etymology of each word, you will discover that although some words may not look like they have a prefix in English, they did have one in the language in which they were born and came into English with it. Not all dictionaries provide etymology, but Etymonline is helpful in this matter.
Thus re- in repeat was a prefix in Latin (same for redundant):
from from Old French repeter, from re- "again" + petere "to go to; attack; strive after; ask for, beseech" (from PIE root *pet- "to rush, to fly").
It is true that *-peat is not an English word before which re- was stuck, the process happened way before it became an English word. So you are not wrong in thinking that it's no use dissecting the word in re-peat. I personally prefer to know that the meaning of the Latin prefix re- is included in this word, because it helps me understand and assimilate it better. This is the same re- you find in remove and recall.
Substance is an interesting word. The word stance was so fused with the prefix sub- that it is hard to say it contains a prefix. However, you are left with stance after removing the prefix, and stance is an English word. Etymonline says it comes:
from sub "up to, under" + stare "to stand," from PIE root *sta- "to stand, make or be firm."
Substance came into English as a philosophical and theological term:
The Ancient Greek term ousia was translated in Latin as essentia or substantia, and hence in English as essence or substance. (Wikipedia)
Substantia was understood like something firm under your feet that helps you stand and not crumble into nothingness. This meaning was not preserved in English.
It is in the 14th century that the meaning of the word shifted to what we are more familiar with today:
Meaning "any kind of corporeal matter" is first attested mid-14c. Sense of "the matter of a study, discourse, etc." first recorded late 14c. (Etymonline)
I would argue that if you use substance in philosophical or theological contexts, you'd better be aware of the prefix sub-. If you use it with its wider meaning, then the prefix is less important and may be ignored. I would personally still consider it a prefix whose meaning is obscured in modern use, but was relevant in the past.
In his dictionary Ologies and Isms: Word Beginnings and Endings, British etymologist and writer Michael Quinion says:
Some examples where sub‑ became attached in Latin, and in which it has a figurative association in English, include subdue (ducere, to lead or draw); sublime (in which the second element may be related to limen, threshold); subscribe (scribere, to write); subjugate (jugum, yoke); submit (mittere, to send, put); and subsequent (sequi, to follow).
I would include substance in this category.
