The first sentence is the much more usual, and it expresses future in the past - an action that is future to the time being described, but not necessarily to the time when we are speaking of it.
Both before and after the party, B would say "A told me that he would come".
Future in the past isn't limited to indirect speech. One other common use is with verbs that express a relationship to information: "I knew/thought/deduced/forgot that he would come".
The second sentence, "he told me that he would have come" is less common, and can mean a few different, rather complicated things.
One would express the perfect completion of an action that's in the future to the time being described:
A says: "I will have (already) come by dinnertime."
B reports: "A told me he would have (already) come by dinnertime."
(It's irrelevant whether dinnertime has already arrived.)
Another is epistemological, as in "Where were you on October 16th?" - "That day I think I would have been at home" (="I'm not certain where I was, but I'm guessing, based on my usual schedule or something"). So:
A says: "I would have come by that time"
B reports immediately: "A is telling me that he would have come by that time."
If you need to move this one step into the past, there is nothing that can modify the would further (not "had willed" or anything like that), so it stays:
B reports later: "A told me that he would have come by that time."
Another is conditional:
A says: "I would have come if the weather had been better".
B reports in the moment: "A is telling me right now that he would have come if the weather had been better."
How would B report this later? A pedantic way might be to try to construct something like: "A told me that would have had come if the weather had had been better." Regardless of whether this is technically grammatical (I'm honestly not sure), certainly nobody speaks or writes that way. The real way B would report it later would leave the conditionals the same as when reporting in the moment:
B reports: "A told me that he would have come if the weather had been better."
There are more ways to use "told me that he would have come", but they are increasingly more complicated and obscure.