What's the difference between "could not have been" and "must not have been"?
- That could not have been an easy task.
- That must not have been an easy task.
I've seen both used. What's the difference?
The two sentences display the epistemic senses of two modal auxiliary verbs: must and could.
Epistemic meanings are abstract and refer to logical predictions and conclusions:
This might/must/could/should/may/will/would be the place.
Deontic meanings are social and have to do with obligations, permissions, and prohibitions:
She may/can/should/must go to the ball.
Negation works differently with modals in their epistemic and deontic senses.
As for the particular pair of modals in the OQ, they interact with negation identically;
epistemic could not/must not turns out to work the same way as deontic may not/can't above.
There's no difference in meaning between the two; there's just a difference in form and sound that some speakers may exploit for their own purposes. Some speakers, in fact, might come to believe that the way they use them is the correct way. But there are many ways, and lots of them work.
Like I said, modals are complex.
More context would be useful here, but personally, I'd be more inclined to use "could" if talking to person B about something person C did, and to use the construction with "must", when talking about the same task to person C. But it's mostly a matter of style, and I"ll have to think about it to come up with a better answer.