It seems to me that semicolons are rarely used today in ordinary English writing - even in newspapers and books. They appear to have been replaced, in many cases, by em-dashes and hyphens (the hyphen being used on the Internet and/or in informal communications). In some cases, semicolons have been replaced by periods (i.e. something which might have previously been one sentence would now be written as more than one sentence). Over what period of time did this change take place, and why?
I'm going to go out on a limb and suggest, on no more evidence than my own inattentive impression, that @Billare is right: your premise is wrong.
Pages 143 ff. of this show that dashes, ellipses and the like were more common in the past than you might think. But they showed up most often in relatively ephemeral works—those which are least likely to come to modern attention. The same is true today: dashes and short sentences are characteristic of informal writing.
It is true that "formal" writing is somewhat more casual now than it used to be; this has always been true, in every age. But seems to me that most of the difference you observe reflects the facts that a) more informal and downright casual literature is available to your inspection than ever before, and b)it is not clearly distinguished by medium from formal writing. Works of philosophy, scholarship, science, and 'high' literature come to you now from the same source and in the same format as news, chat, personal correspondence, and light or sensational reading.
So you're now averaging "formal" and "ephemeral" together, which artificially diminishes the relative frequency of characteristically "formal" punctuation.
It fits in with a longer-term trend toward shorter sentences and plainer writing. Semicolons are only useful in long sentences.
I am not sure why this happened. On the one hand, as Martha F.’s answer indicates, good writers these days are keenly aware that
and perhaps it is as a consequence that sentences like this one, from A Tale of Two Cities:
are now rarely found in nature.
But I think that view is not quite adequate. There’s really nothing particularly taxing about this thirty-clause pile-up, no reason you couldn’t stop halfway through, set it down, and read the rest tomorrow. It’s just that it would be so much more manly and Hemingway-ian with full stops. A matter of changing tastes.
The semi-colon has not been entirely supplanted. For example, a semi-colon is frequently the correct separator for a list, while a dash never would be.
The short answer, I think, is that the em-dash makes it easier to read the sentence and thus more likely that the reader will understand your point and want to continue reading.
Readers today are inundated with far more to read than ever before, and any structure that forces the reader to do more work will be detrimental to their likelihood of reading further.
Grammatically, semicolons are used to connect two independent clauses without the use of a linking preposition. However, the type of connection is not stated outright; the reader is supposed to assume a reason for not separating the clauses into sentences. (In the last sentence, the implied connection between clauses could be expressed by the word therefore.)
When a writer uses an em-dash rather than a semicolon, there is white space around the connection. White space, particularly for visual learners, signals that there is some work to be done by the reader to understand the connection between two things. It also allows metaphorical space for that to happen, since it forces the reader to pause and make the connection.
A semicolon doesn't offer that same amount of white space. It is a subtle cue, and can slow the reader down if they don't see the period above the comma. (Anything that makes a reader pause and say "huh" will increase the likelihood that they will not continue to read.)