If he wasn't going on the camping, he would stay home. (he might go to the camping) If he weren't going on the camping, he would stay home. (he went to the camping and didn't stay home.)
The first is a real condition, and the second is hypothetical because of was and were. but if the subject is "they", both will be the same. So how to know which is the hypothetical or real condition?
"If" is the thing that is making it hypothetical, so both your examples are hypothetical. However, you would say "going camping", not "going on the camping"... or you could say "going on the camping trip".
The first one is how Americans would say it: "If he wasn't going on the camping trip, he would stay home." The second is how UK-influenced countries use it: "If he weren't going camping, he would stay (at) home."
The real condition would be "He is not going camping, so he will stay home."
I don't think your distinction between 'real' and 'hypothetical' is a valid one, and your examples are not idiomatic English.
If the person did go camping, you would say "If he had not gone camping, he would have stayed at home." A hypothetical condition would be "If he were not going camping, he could have accepted the invitation to his friend's party."
In "He didn't know if they were going camping", the thing that's "hypothetical" is his knowledge of their intent. The statement as a whole (that he didn't know) is not hypothetical (in a linguistic sense).
a. If they weren't going camping, they would stay home. [hypothetical]
b. If they weren't going camping, they stayed home. [real]
You have to look at context to determine whether it's hypothetical or real.
