Can "You mustn't have done that" have a similar meaning to "You shouldn't have done that" / "You were not supposed to do it (but you did)"? (not logical probability but obligation)
Since we have the imperatives like "Have done with that!", wouldn't it be possible to use must to express prohibition of a future action: "You mustn't have done that (by the time the bus reaches here tomorrow)"?
"mustn't have done" is definitely not equivalent to "shouldn't have done." The latter is used when you actually did the thing and the speaker is either commenting or rebuking you. The former is most often used as a statement or conclusion. Example: you state that you locked the door this morning, but someone else observes people opening the door sans key, and thus points out to you that "you mustn't have locked the door."
Your second question contains a construct which might be legal (I'm not quite sure), but certainly is very awkward either to say or write. I would go with either "You must not do that (by the time...)" or the somewhat stronger "you shall not have done that..." .