In formal writing, should I use an em dash
The Earth consists of three layers — crust, mantle, and core.
or a colon
The Earth consists of three layers: crust, mantle, and core.
As @tchrist said, I would recommend the colon: "The Earth consists of three layers: crust, mantle and core." However, the comma before the conjunction in the list is optional, as I understand; see http://en.wikipedia.org/wiki/Serial_comma.
From some quick research, it seems that the colon generally introduces a continuation of the thought or a list of items: more information that the writer is including. The em-dash, on the other hand, signals a change in topic (a parenthesis) or a summary of the preceding thought—a sudden movement of thought which should not be read as a continuation of the first part of the sentence.
As an example, here is a sentence from Charles Dickens' Great Expectations which uses both the colon and the em-dash:
Notice how the sentence thought continues after the colon, while the em-dash sets apart a thought that can be omitted without damage to the original idea of the sentence: "staring distrustfully...and often stopping...to listen."