Those are sentences with elliptic clauses, which means something is left out that has to be supplied:
That's what I was thinking, but [what I] couldn't articulate—the "or" being inclusive...
I tried, but [I] couldn't do it.
I tried and tried, but [I] couldn't.
The conventional stylistic rule of thumb here is that independent clauses are separated by commas, while elliptical clauses are not. So at the very least you could write those sentences without the comma.
But is the comma permissible anyway? That is a more nuanced stylistic question. It may depend on which style book you ask. I believe they will generally tell you that you shouldn't use a comma with an elliptical clause unless you have a special reason to do so.
This reason could be that you want to introduce an extra long pause to emphasise the contrast of a but clause. Or that the elliptical clause should be seen as a kind of afterthought: afterthoughts are generally separated by commas. Or that you have more than one elliptical clause, and you want to indicate that the first one is strongly conexed to the main clause while the second is more separate. Or that the sentence is very long already and would become more readable with an extra comma. Or some other reason that applies to non-compulsory commas in any other position.
- That's what I was thinking, but couldn't articulate—the "or" being inclusive...
In your first example, I would probably not use a comma, because it really isn't necessary and the sentence sounds nicer, more concise, without it. But you might feel you want to emphasise the contrast using an extra special pause. It's ultimately up to you.
- I tried, but couldn't do it.
In your second example, I agree with you that the comma is the least appropriate. One reason is that the sentence is short, and shorter sentences generally have fewer non-compulsory commas. Another is that it is easy to 'fix' the sentence if you really need the comma, by adding I (the subject) to the second clause, thereby removing the ellipsis; this results in two independent clauses, which normally have a comma between them. The fix is not as easy in the first example, because you would need to add two words; that can be a reason to do it the other way.
- I tried and tried, but couldn't.
Here we have three clauses, the last two elliptical. I might very well use a comma here, to indicate that the first two clauses belong together and the third is separate. It would actually be functional here to so mark the degree of conexion.