When we say "Our team worked hard", then is it expected to refer the speaker + his own team OR the speaker + listener + their team together? Because in both these cases OUR is common word to be used!
The ability to distinguish between the inclusive we and the exclusive we is referred to as clusivity and is a feature lacking in English. Without additional context, you're correct that it's impossible to determine whether you intended to include the listener in the pronoun or not, although in many cases knowledge of the basic scenario you're describing can be enough to clarify (e.g. if you and the listener were never on a team together or at least a hard working one). However, should it fail to be clear based on such knowledge you're pretty much stuck either adding in a clarifying phrase or simply avoiding the use of our by spelling out who you mean.