Most words only have one correct negative prefix out of "in-", "im-", and "un-". Why are both "imbalance" and "unbalance" both grammatically correct (but "inbalance" is not")? What are differences in meaning and usage of these two opposites of "balance"? Can they be regarded as synonyms of each other? Are there any other examples of such co-existing variants with different uses?
About im-, etymoline explains:
variant of in- before -b-, -m-, -p- in the sense of "not, opposite of"
Bear in mind that in- comes from Latin, and so it will follow Latin phonetic assimilation:
In- is a word-forming element meaning "not, opposite of, without" (also im-, il-, ir- by assimilation of -n- with following consonant, a tendency which began in later Latin).
Therefore, before b, m or p the prefix in- will always become im-. 1
As for un-, the same Etymoline says it is a:
prefix of negation, Old English un-, from Proto-Germanic *un-.
Un- never becomes *um-, because it is not a Latin prefix.
Etymoline goes on to explain the difference in use of un- and in- (and its variants):
Un- is the most prolific of English prefixes, freely and widely used in Old English, where it forms more than 1,000 compounds. It underwent a mass extinction in early Middle English, but emerged with renewed vigor 16c. to form compounds with native and imported words. It disputes with Latin-derived cognate in- the right to form the negation of certain words (indigestable/undigestable, etc.), and though both might be deployed in cooperation to indicate shades of meaning (unfamous/infamous), typically they are not.
1 I underline that when it is a prefix, it changes into -im. You do find such occurrences as input (where in comes from an adverb). So do not confuse inpatient (a patient who lives in hospital while under treatment.) with impatient (not patient)!
As for the difference in meaning between imbalance and unbalance, imbalance seems to refer more to a lack of balance in quantity or quality among the components of a whole, whereas unbalance can refer to the lack of (physical or mental) equilibrium.
About imbalance,Vocabulary.com says:
An imbalance occurs when you have too much of some things and too little of others. If you put so much pepper in your soup that you can't taste the other spices, then you caused an imbalance in your flavoring.
About unbalance, the same dictionary gives two first definitions to it as a verb, but it does also record its use as a noun:
To unbalance is to make unsteady or uneven. A sudden gust of wind might unbalance you when you're on a sailboat — don't forget to wear a life jacket!
Throw something off balance, and you unbalance it. Tickling someone who's in a one-legged yoga stance will unbalance them, and leaning too far to one side will unbalance a new bike rider. You can also call this state of being out of kilter or wobbly unbalance: "The unbalance of the tightrope walker made me glad there was a net underneath him!" And figuratively, unbalance can also mean "disturb or upset."
Now I might "alarm" you even more, if I tell you that there is a third antonym of balance: disbalance, but you can be at peace, it is rarely used (with a particular connotation of disturbed/upset balance).
Concluding, I will just add this GNgram that shows how the three terms are compared in use: imbalance is the most common.