The difference is that because is a statement of causality, while in that is used to describe a relationship or a perception.
The car's steering was off because an accident had bent one of the tie rods.
The car's steering was off in that I could feel a slight wobble even when driving on level, flat pavement.
In the second example, you wouldn't use because because the symptom is not the cause. In the first example you wouldn't use in that because the bent tie rod is a direct cause, not a perception (unless it involved speculation, which here it does not).
I think of in that as a shorter form of in the sense that (or to the degree that) something appears to be a certain way.