What is the author trying to convey with the word 'O' in the following:
He has told you, O man, what is good;
This question is too basic; it can be definitively and permanently answered by a single link to a standard internet reference source designed specifically to find that type of information. See the FAQ for guidance on how to improve it.
This is the vocative case:
Modern English does not have a formal vocative case, and may use the nominative to fill that role.
The cases are more apparent when looking at some other languages like Latin. A well-known passage from Alice's Adventures in Wonderland where Alice speaks to a mouse reviews them, ending in the vocative:
Note that the "O" itself can be considered an interjection, much like "Oh" (thanks, @Kit and @Cerberus!); it might be more clear to say it is an interjection accompanying a noun that is in the (unmarked) vocative, "man" in your example.
'O' is an archaic word used to emphasize the subject being addressed. In your example (Malachi chapter 2 IIRC) the speaker wishes to address "man," so he says "O man," not to be confused with the modern idiom "oh man."