Both constant-factor and competitive can describe an algorithm separately and in other compound adjectives (e.g., "constant-factor approximation or "throughput-competitive").
So, you can punctuate them as either "constant-factor, competitive algorithm" or "constant-factor competitive algorithm" though. Some examples of both comma and no comma algorithm titles in published articles: here, here, and here.
Hyphen use on the first word indicates it is compound and comma use between your adjectives just separates them, of course. Using both simply adds to the readability of long titles like these.
See Apoorva's answer for why you might want to separate with a comma:
(1) Can you replace the comma with the word and?
(2) Can you reverse the order of the adjectives and keep the same meaning?
If the paired adjectives fail the two-part test, then no comma is
used. This shows that they must remain in a certain order to make
sense. These are called cumulative adjectives.
If two adjectives modify a noun in the same way, place a comma between
the two adjectives. These are called coordinate adjectives.