The dictionary describes racism as:
a belief or doctrine that inherent differences among the various human
racial groups determine cultural or individual achievement, usually
involving the idea that one's own race is superior and has the right
to dominate others or that a particular racial group is inferior to
Using the definition above, merely making a comparison between the color of coffee and the color of skin of the predominant (black) population in Africa is not technically racist if there is no inference that people of black skin color are inferior in any way to people of other skin colors.
However, that said, in recent history (the last few hundred years) there has been so much negative connotation associated with mentioning black skin color that it becomes nearly impossible to mention black skin color without causing any typical listener to immediately recall in his or her subconscious thousands of negative stereotypes or other comments that he or she has heard in the past associated with black skin color. Therefore, one could argue that it is impossible in modern society to mention black skin color without at least some (even unintended) negative association, if merely from the history of negative associations.
Compare, for instance, the strong movement against the team name Washington Redskins. This is really on point to your question. Can the mere mention of a group of people (in this case native Americans) by their skin color (red - ie redder than Europeans but not so black as Africans) be considered racist? There are a lot of people who think so.
You therefore can conclude, factually, that while the strict dictionary definition will not prove that a mere reference to Africans as black is racist, a lot of people will interpret that sort of comment as racist for the reasons I and the articles above have noted. And if you know that many people will interpret a comment as racist, then it becomes hard for you to claim you are able to say it without it being racist, since you know, empirically, that it will likely be interpreted that way by a large number of people. Your own empirical evidence also supports this conclusion.