Is there a word that means doing the right thing for the wrong reason or getting the right conclusion from the wrong set of presumptions?
I think this question can't have a clear answer because this phrase is quite open.
Think about all the situations where this could be applied.
So going to the root of the problem, I think what this phrase is expressing is that the person isn't aware of all the circumstances (or has the wrong idea) when doing something.
So I agree with @Andy that it might be unwittingly. Or in my opinion inadvertently would be a closer fit to.
That said, it might also be expressed unintentional, since he/she is doing something wrong, thinking its the right thing.
But all those meaning don't cover the harshest way to use this phrase that would make the "accused" a naif, confused or misguided (I like this last actually) since you could be meaning: "although you are doing the right thing, there is not much merit to it, since it's almost by chance, you don't have a philosophy behind your actions (or a wrong one)"
The word "Lucky" is frequently used in that situation.
For example, in a case where someone confronted with two doors, decorated with writing in a foreign language, selects the one that is on the right as the "entrance" on the basis that everyone is right handed. He has made a correct selection, based on incorrect information (that is not why the door is the entrance). One might say to him:
"Wow, so you just happened to pick the correct choice despite having faulty reasoning? What a lucky break!"
Although now that I'm thinking about it, the word "Fortuitous" is closer to what I meant.
Lacking good sense or judgment; unwise.