OK I see you got this answer in comments but I'll still make an answer out of it because I'm not sure the existing ones address this point.
Basically, beyond the meaning there is a grammar issue. The main reason you can't swap "end up" and "end in" in the following sentences and that the result "looks wrong":
Her marriage ended in divorce.
You will end up being fired.
... Is that "end in" and "end up" don't fit in the grammar the same way.
I guess maybe one way of describing the difference could be: X ends in Y is used when X describes things that can have an end (like a marriage), and Y describes the end that X had. In that situation the subject of "ends" is the specific thing that has an ending, and Y is a noun or a noun-equivalent that refers to the ending. X ends up Y is used when X is any process or thing at all that changes over time, and Y describes the final situation X attained after the changes the sentence is about. In terms of the constraints on what Y can be grammatically speaking I think one way of understanding it might be that Y has to be a phrase that X could be the subject of. In other words when you say "X ends up Y", grammatically you ought to be able to mentally substitute "Stuff happened, now X is Y".
Her marriage ended in divorce.
The subject is "her marriage" - something that we know can end. It's not "her"; marriages can end, people don't. Well, grammatically at least. And "divorce" is a noun that describes one of the possible ends marriages can have. You couldn't put "end up" in that sentence for the same reasons you couldn't say "Stuff happened, now her marriage is divorce". First, "divorce" can't be a noun in that sentence it would have to be a past participle, and even if it were marriages don't divorce, people do. What you could say is:
She ended up divorced
Here "she" is the subject, not the marriage, and it's an appropriate subject you can apply the word "divorced" to. "Divorced" describes the situation she ended up in.
I think it's actually really interesting what you can do with this difference, because like Barmar in comments I thought that the reason you couldn't say "her marriage ended up divorce" was that "divorce" is a noun, and I figured there was just a general rule that "end up" cannot be followed by a noun. However "X is Y" can have Y be a noun. So can we apply that to "end up" ? And I think the answer is yes, here is an example:
After the alien attack killed off most of the government she ended up President
I think in this case we can sort of see a hidden "becoming the President" lurking in that sentence, but it's still a noun and I think the sentence is one you could realistically see out in the world that wouldn't strike people as incorrect. I'd guess the reason "end up" almost never ends in a noun is that it's just rare for a noun to describe a situation you could fit into a sentence of the form "This subject is [situation]".
If I can apply this to two example sentences from another answer:
If you keep driving like that, you'll end up in an accident
This is a perfectly good sentence and also fits the "stuff happened now X is Y" model, we can say "you were in an accident". The noun "an accident" wouldn't work in this phrase, it would essentially saying the person would become "an accident". I think this would work if we replaced "an accident" by something a person can be, like "a cautionary tale". But even then I think it would be a bit elliptical and awkward, I think most people would say "you'll end up as a cautionary tale" or "you'll end up being a cautionary tale". Basically I'd guess that even when a noun technically works, it isn't preferred in this construction, maybe because it's harder to parse what exact situation is being described.
However I don't think this sentence is quite as fine, although I don't think it wouldn't be that shocking to hear said
If you keep driving like that, you'll end in an accident
What this says is that "you", as a process, will end and that the ending will be of the "in an accident" variety. Which technically is a completely accurate description from a biological perspective but I don't think it's quite made it into English grammar outside of certain cute phrases like "I will end you". I think most would find it more natural or correct to say something like "your life will end in an accident" or "it [i.e. things overall with you driving like that] will end in an accident".
But to tell the absolute truth I bet if a person says "you'll end in an accident" they don't mean "you'll die" and aren't actually using the end-in form at all, they're trying to say "you'll end up in an accident" and just elided the "up", hence me saying that I don't think it's quite fine but also isn't shocking. Compare with a phrase that doesn't have "in", like "if you drive too far you'll end at the beach". I think it's both sloppy and non-shocking in the exact same way as "you'll end in an accident" is but it's obvious isn't a specific construction different from "end up" being used there.
Unlike "end up" that seems to have flexibility for what comes next even if it's almost never a noun, I think "end in" might always end with a noun or phrasal equivalent. For example:
This will end in tears
This whole situation ended in her breaking her vows and moving to Canada
(even that second sentence would probably be better with "ended with" tbh but I think it's still OK).
We couldn't say "she ended in divorced" because 1) for that to work she has to end somehow, and 2) it really has to be a noun or equivalent after the "end in" there. It's like where "end up" asks for an actual description of a situation, "end in" wants a label. I can't write:
This whole situation ended in she broke her vows and moved to Canada
but my brain still thinks it can rescue that sentence, and the way it wants to do it is by adding punctuation/intonation like so:
This whole situation ended in: she broke her vows and moved to Canada
like this can be rescued by a syntactic break that makes it clear the second half of the sentence doesn't flow from the first, but is playing the role of a label pointing to one specific outcome. You get the feeling such a sentence would only be natural if it's conveying the idea that "she broke her vows and moved to Canada" isn't a random thing she did, but is one of a few specific options that were made clear earlier in the text or conversation which is why it makes sense to write or say the sentence like that.
Maybe a rule of thumb parallel to the one for end up could be to see if you can substitute a different construction that has the same rules. For example: maybe "X ends in Y" works if you could also say "Well, X has ended. How do you feel about Y?".
tl;dr: I think the following substitutions might work for figuring out the grammar around each form. In each case, the "X ends in Y" form should work if (and only if?) the second form is a grammatically correct sentence.
"X ends in Y" => "Well, X has ended. How do you feel about Y?"
"X ends up Y" => "Stuff happened, now X is Y."
Happy to take corrections tho.