— What did you do yesterday?
— I was working.
I presume this is not grammatically correct; however, I often reply in this way (I'm English). Also vice versa: "What were you doing yesterday?" — "I worked."
Does it matter?
I was working is certainly a grammatical answer to the question, but whether you say that or I worked depends on the relationship between the speakers, what has gone on previously in the conversation and on what impression you’re trying to give.
If you are asked What were you doing yesterday? or What did you do yesterday? you won’t often reply I was working or I worked. What were you doing yesterday?, for example, can imply a suspicion, or even an accusation. In that case, the reply might well be What do you mean? I wasn’t doing anything yesterday. Similarly, if you’re asked What did you do yesterday? you might reply Um, let me see. First I went to buy a newspaper, then I took the dog for a walk . . .
There can be no single answer to a question like this. Context determines everything.
There is absolutely no reason to reply to a question with the tense and aspect it was given in. There are indeed times when it would be inappropriate:
Here the mixture of past continuous and simple past is the best way to convey the two events we want to include in the answer.
Even the bare progressive, "I was working," has an advantage over the simple past "I worked," in precisely the way that the progressive serves; It includes the information that this took some time, and in the context implies it took up much of the time frame referred to ("yesterday").
We can even imply the past with the future:
Here we respond with a statement about the future, that both entails an answer about the past (it has not been written yet), and provides useful extra information (whether useful to the querent, useful to the answerer in excusing their not having done the task yet, or both).
And of course, simple yes or no answers have no real tense or aspect:
When we've a choice of answering in a matching tense and aspect, or in another, we're slightly more likely to do so just as a matter of mirroring (we change a lot of how we act to match how those we are conversing with act, including a lot of our language choices), but there's absolutely no grammatical rule.
We wouldn't mirror in this way if differing from the question gave a better answer (by whatever definition of better you choose to apply) and we might deliberately not mirror (e.g. answering in a more or less formal register can hint at a disagreement with the querent without expressing it outright). Choosing to mirror when possible, is a style choice available to us, but only one.
Grammatically, the correct answer for the question "What did you do yesterday?" is "I worked." But in spoken English it does not matter if you answer like "I was working."