The earliest reference I can find is in the OED reference to 'colourful' :
1914 in Hist. N. Otago from 1853 (1978) 117 Shouting colourful epithets at them, and waving a Maori kit which contained her supper.
That quote would appear to bear the meaning of strong language, a 'colourful epithet'.
The NGram for 'colourful language' begins just after that 1914 quote, in about 1918.
'Colourful' bears a general meaning :
b. euphem. Disreputable; questionable; notorious.
so I assume that is the derivation of its application to language.
And, again, that traces back to the First World war period :
1921 K. Tappert Viewpoints in Biogr. 55 In spite of Nell Gwyn's rather colorful life, the English people love her memory.
'Colourful epithet' is BrE and 'colorful life' is AmE and all this appears to begin in the 1914/1920 period so maybe it is an expression of those times of American/British intermingling and postwar relaxation of society.
'Colourful' does not condemn the language being used, as Victorian ideals would have done so. It is a comment, not a criticism. I think it denotes the changes in society occurring during and after the First World War and involving the unprecedented social upheavals which happened at the time.