I can think of a couple of phrases that come close, but don't exactly capture what you're looking for.
To put a Band-Aid on a situation is to make some small, short-term effort to fix something, without addressing the underlying causes.
A way to describe people's feeling in such a situation could be smoldering tension - there's a sense of a deeper underlying issue that hasn't been addressed, which could spark a larger problem. The word smolder here aligns very well with the idea of "fire under the ashes".
Such a situation could be called a powder keg, which is a delicate state of affairs that could escalate quickly and violently with little provocation.
Each of these addresses different aspects of the situation, but none of them individually completely captures the act of making a short term fix that might be worse in the long run.