It's grammatically incorrect: "You know you're having a rough day when kittens don't even make you smile" is what was meant. It was either a typo, a thinko, or just another ignoramus expressing itself.
That said, however, it is possible to say something like this:
Your having a bad day is not going to ruin my day.
Some native speakers might write and say it this way:
You having a bad day is not going to ruin my day.
Now that you've added a question, I'll add more of an answer.
If you want to make the first sentence grammatical without changing "your" to "you're", it has to be something like this:
You know (that) your having a rough day is true when kittens don't even make you smile.
The syntax of the sentence is different. In the sentence immediately above, your having a rough day is a gerund clause that serves as the subject of a subordinate clause because it is preceded by that, what Linguistics Professor John Lawler calls a complementizer, which means that it requires a predicate: a verb and what some people would call a subject complement. I don't want to dwell on terminology, because it isn't always agreed upon, even by professional linguists (I'm not a professional linguist).
In your original example and your added analysis, the verb and subject complement are missing, but it's not grammatically possible to do that in this case by eliding them from the sentence. If you had given us this instead:
Your having a rough day is clear. You know it when kittens don't even make you smile.
then you can say "it" = "your having a rough day", because it is a pronoun that refers back to the first sentence.
The gerund clause is a noun phrase. If I substitute another noun phrase, e.g.,:
You know giraffes when kittens don't even make you smile.
you should be able to see that something is wrong because the sentence is incomplete, not elided. I can change it to:
You know giraffes are evil when kittens don't even make you smile.
Now it's a grammatical sentence, but it's semantically unacceptable because it's totally illogical.
To make the other new sentence simpler, let's change it to this:
He resents (you/your) being more popular.
or
He resents (you/your) having a good day.
If:
A: I'm having a good day.
B: I know it. (it = you are having a good day) But John resents it. (it = the fact that you are having a good day)
I can say:
1: I resent your being more popular.
just as I can say:
2: I resent giraffes.
but I must say:
3: I resent (it) that you are more popular.
