In formal writing, should I use an em dash
The Earth consists of three layers — crust, mantle, and core.
or a colon
The Earth consists of three layers: crust, mantle, and core.
As @tchrist said, I would recommend the colon: "The Earth consists of three layers: crust, mantle and core." However, the comma before the conjunction in the list is optional, as I understand; see http://en.wikipedia.org/wiki/Serial_comma.
From some quick research, it seems that the colon generally introduces a continuation of the thought or a list of items: more information that the writer is including. The em-dash, on the other hand, signals a change in topic (a parenthesis) or a summary of the preceding thought—a sudden movement of thought which should not be read as a continuation of the first part of the sentence.
As an example, here is a sentence from Charles Dickens' Great Expectations which uses both the colon and the em-dash:
He was gobbling mincemeat, meatbone, bread, cheese, and pork pie, all at once: staring distrustfully while he did so at the mist all round us, and often stopping—even stopping his jaws—to listen.
Notice how the sentence thought continues after the colon, while the em-dash sets apart a thought that can be omitted without damage to the original idea of the sentence: "staring distrustfully...and often stopping...to listen."