I also believe that "time" is the correct choice. It is not a "count noun" that could be pluralized, as the word "chairs" in "You have broken both of our chairs." "Times" is a conventional word, however, in certain expressions, as in "These are good times." In that sense, "times" can be thought of as a count noun as in "These are good days" "Or those were favorable periods [of time]." But in your question, I would think of "time" as I would think of the word "money." One could say "You're wasting both of our dollars" (for example, if each person had a dollar bill and each thought that you were spending them on something frivolous), but one would not say "You're wasting both of our monies." (Though, again, there are certain contexts in which "monies" would be correct.) But as the character did not specify that the other was wasting something countable, like "hours," then the choice should have been "time." See https://owl.english.purdue.edu/owl/owlprint/541/

I also believe that "time" is the correct choice. It is not a "count noun" that could be pluralized, as the word "chairs" in "You have broken both of our chairs." "Times" is a conventional word, however, in certain expressions, as in "These are good times." In that sense, "times" can be thought of as a count noun as in "These are good days" "Or those were favorable periods [of time]." But in your question, I would think of "time" as I would think of the word "money." One could say "You're wasting both of our dollars" (for example, if each person had a dollar bill and each thought that you were spending them on something frivolous), but one would not say "You're wasting both of our monies." (Though, again, there are certain contexts in which "monies" would be correct.) But as the character did not specify that the other was wasting something countable, like "hours," then the choice should have been "time," and I would have said it as "You're wasting both our time." See https://owl.english.purdue.edu/owl/owlprint/541/