I know that you can say "This is the one that is broken," or "This is my car, which is broken," but as written it seems odd. Is it wrong? My theory is that a restrictive clause cannot be used in combination with a possessive adjective unless you're referring to a person (E.g., "This is my nephew who is sick," would be okay.)
There's nothing grammatically wrong with it, but..
To me, it seems like you are clarifying that this is your car that is broken, as opposed to your other car, which is not.
However, I think it sounds a bit off, and could use a rephrasing. Like the ones you mentioned.
Your theory is correct. If you modify a noun with a possessive adjective, there is usually only one thing/person you could be referring to; in that case, any more restricting would result in some redundancy. Or it could be the other way around: because a restrictive clause already restricts what a noun refers to, a possessive adjective won't be necessary any more. In most cases either one or the other is redundant and thus awkward.
Even in this case, my could and should be left out, because it isn't really necessary. The article the will do; my has stronger referential value than the article and would cause some redundancy. Even so, my wouldn't be that bad here; you'll probably see it used informally now and then. As an alternative, the relative clause could be made non-restrictive, with which and a comma.