TL;DR: "Midnight" should only be used where the night is certain or conceptional. Conceptionally the individual night does not matter. In denoting a specific point in time, clarify the night (not just the day) - especially when on a deadline.
Colloquial use can be a strong argument if a term stands firm in the language. If you are reading this then the term does not stand firm. Ultimately, you will have to observe or ask others how they use "midnight [of xday]" - which is how colloquiality arises. Colloquere, colloquere!
Repeating some thoughts others uttered in order to elaborate
"AM" stands for "ante meridiem" and "PM" for "post meridiem", meaning before and after midday respectively. "Midday" relates to a certain or conceptional day, not a night. In the same way, "midnight" relates to a certain or conceptional night, not a day.
While definition-wise midnight should be neither AM nor PM, from philosophical and technical points midnight is both preceeding and succeeding (some) midday thus being both AM and PM; dito midday.
Binding midnight (and midday) to AM or PM and still avoiding ambiguity requires understood convention which works in small, agreeing and identified groups only and so makes for bad use in wider language where the group and convention of a speaker is not easily identified.
It may be more useful to have 12 CM (contra meridiem, opposite midday) for midnight and 12 HM (hoc meridiem, this midday) for midday - relating to conceptional terms rather than any specific midday or day.
Midnight is written as "12am" which would imply that it's in the morning. Therefore, it should be at the start of the day.
If we apply your argument to 12pm, being labeled in that fashion implies that it is in the evening, and therefore it must be at the end of the day. Thus, people are wrong to go around calling it "noon", which is clearly in the middle of the day, not the end.
So while "12am" may claim to be before midday it may just be treated as a synonym to "12pm" (marrying the arguments of Damovisa and Hellion) - with both designations being utterly useless to disambiguate except for standing convention. So the "12" may be more significant. I do not know of any clock (as opposed to what is called a "timer" - ugh), calendar or other time-giver counting time down. Assuming a count-up, "12" has come after some "11" and something has been counting up a dozen times. That rather relates "12am" to whatever happened before. This could argue for midnight belonging to the day before.
Mr. Shiny and New 安宇:
However, for convenience, most people lump the 12:00:00 time with its nearest neighbour, 12:00:01, which IS AM or PM.
There could be an argument for convenience here if "nearest" was effectually defined. Time-wise, 11:59:59 (or 23:59:60) is no further away and offers no less convenience.
In conversation, the 'night' of which 'midnight' is in the middle, is considered the night of the date mentioned.
This replaced the ambiguity of "X's midnight" with that of "night of the date". This can only help if the night is sufficiently specified by the date which may not be the case.
(Y)Our mission, syctai, is to clarify the night before using "midnight" - unless speaking conceptionally.