The word itself like any other noun or pronoun, can be informed by a range of prepositions, each of them conveying its own meaning.
"The door opened of itself", may be an older idiomatic form, not much in current use.
The door opening is no different whatever to the machine which "started by itself".
You could say "the camera took a picture of itself" (assuming there are cameras capable of such things), "the company posted a letter to itself", "the bank accepted a charge on itself", "the group entered into a dialogue with itself".
There is a separate idiomatic form of "of itself", where we say such things as "the suggestion was not of itself helpful, but it did lead to a useful debate". But in this situation one could equally well use by itself.