Why do we say my goodness? It doesn't sound appropriate for the contexts in which it is used. Both the my and the goodness don’t seem to bear on a surprising or startling situation. When was it started, and why?
The Phrase Doctor writes that my goodness is a minced oath:
My goodness --> My God
My gosh --> My God
So the phrase was used instead of swearing, with goodness being a euphemism. The Oxford English Dictionary adds that goodness! has an established meaning in this vein, writing:
In various exclamatory phrases, in which the original reference was to the goodness of God (cf. sense 2a above), as goodness gracious!, goodness (only) knows! , †for goodness!, for goodness' sake!, in the name of goodness!, (I wish) to goodness! , surely to goodness!, thank goodness!, etc., or simply goodness!
The first written usage is by Shakespeare, who wrote:
1623 Shakespeare & J. Fletcher Henry VIII Prol. 23 Therefore, for Goodnesse sake, and as you are knowne The First and Happiest Hearers of the Towne, Be sad, as we would make ye.
There isn't a notation about the first written use of my goodness, but it definitely uses the sense of goodness which has been around since the 1600s.