Please note that this is not based on any cited source but on my own observations.
Colloquia and seminars both happen in an academic setting. At my university we have a weekly physics colloquium that — in general — is geared to a well educated, but not specialized audience. (I.e., a particle physicist will present a topic on a fairly advanced level, but so that, say, a condensed matter physicist will still be able to understand). In general they seem to be more "populist" and less technical, covering popular topics in physics (quantum information, graphene) and in the news (global warming, nuclear weapons/power) but from the perspective of a scientist.
A seminar on the other hand, in an academic setting, is a much more specialized meeting, also with a formal academic presentation. For instance, there are weekly seminar meeting for the Atomic, Particle, String Theory, Condensed Matter, and Astrophysics groups. There is also an invited speaker, but the audience is much more technically versed and the topics tend to be much more technical or specific to the field. Generally someone from outside the field will have trouble understanding a seminar presentation.
In a grand sense the two words are equivalent, but a colloquium, as pointed out, is literally a "conversation" and in general has a connotation of being more broad, more accessible, or on a more popular topic.