Looking at the definition here,
con·sort (knsôrt) n.
1. A husband or wife, especially the spouse of a monarch.
2. A companion or partner.
3. A ship accompanying another in travel.
4. Partnership; association: governed in consort with her advisers.
5. A group; a company: a consort of fellow diplomats.
6. Music a. An instrumental ensemble. b. An ensemble using instruments of the same family. v. (kn-sôrt) con·sort·ed,
con·sort·ing, con·sorts v.intr.
1. To keep company; associate: a politician known to consort with gangsters.
2. To be in accord or agreement. v.tr.
1. To unite in company; associate.
2. Obsolete a. To escort; accompany. b. To espouse.
there doesn't appear to be anything negative about the word. I suppose its appearance in the phrase consorting with known criminals might give the impression that there is something disreputable about consorting, but don't forget usages such as "the Duke of Edinburgh, Prince Consort", which is quite a prestigious title, I am led to believe.