In some countries where there is serious shortage of skilled personnel or employable persons, it is common to describe the situation as a problem of unemployability. However, this word still appears to be unrecognized by most if not all dictionaries. Is there some better word or does it break any rule? Why has it been unaccepted for a long time?
The Google Ngram Viewer is often useful to see the past popularity of words over the decades. Sometimes it's understandable - concerns about un/employability jumping up during the depression years, declining during the Second World War and the stability of the 50s. Sometimes it's no so obvious - why has 'employability' been recently booming, while other variants are in decline?
Having said that, different dictionaries have their own standards as to when/why a word will get added or even left out. "unemployability" may get used, but certainly not as often as various other words, and no dictionary can include every single word in the English language.
The OED is probably the most extensive dictionary, but even then:
The Second Edition of the 20-volume Oxford English Dictionary contains full entries for 171,476 words in current use, and 47,156 obsolete words. To this may be added around 9,500 derivative words included as subentries. [...]
This suggests that there are, at the very least, a quarter of a million distinct English words, excluding inflections, and words from technical and regional vocabulary not covered by the OED, or words not yet added to the published dictionary, of which perhaps 20 per cent are no longer in current use. If distinct senses were counted, the total would probably approach three quarters of a million.
@Neil's Ngram link suggests that usage of "unemployability" may not be enough to guarantee it a place in each and every dictionary.
In general, using a word that does not appear even in any dictionary does not "break any rule". One of the reasons a word does get added is because of the ways in which and how often it is used - people were quite happily using "badassery", "selfie", and "twerk" before they were added by Oxford.
A word exists before it gets a dictionary entry - not the other way around. Dictionaries do not prescribe, but rather record, usage.
If you will be understood when you use a non-dictionary word, then feel free.