The phrase "crème de la crème" means to be the best of the best. Is there a phrase that means the opposite of this, that is, to be the worst of the worst? The phrase doesn't have to come from French.
Acceding that 'Bottom of the Barrel' is a good and sufficient answer (suggested by @JohnLawler), if we were to attempt to mirror the sense of the expression 'crème de la crème' (the separated 'best' part of a product) we might arrive at 'dregs', or 'scum'. which both refer to the separated 'worst' part of a product.
'Dregs' as the OED advises us is:
dreg ▪ I.dreg, n. Chiefly in pl. dregs (drɛgz). Forms: 3–4 drege, 4– dreg, (6 dredge, 7 dregge); pl. 4–7 dregges, (5 -is, -ys, dregys, 6 dragges), 6–7 dreggs, 6– dregs (6–7 drags, dredges). [Probably from Norse: cf. Icel. dreggjar pl., Sw. drägg pl. dregs, lees.] 1. (Usually pl.) The sediment of liquors; the more solid particles which settle at the bottom of a solution or other liquid; grounds, lees, feculent matters. Also fig.
It has a long history of use in the technical sense:
a 1300 E.E. Psalter lxxiv. 9 [lxxv. 8] Drege in him [v.r. his dreg; Vulg. fæx eius] noght is littled; drinke sal al þa sinfulle. 1377 Langl. P. Pl. B. xix. 397 Whil I can selle Bothe dregges and draffe and drawe it at on hole, Þikke ale and þinne ale.
...but also a long history in a derogatory sense:
- fig. The most worthless part or parts; the base or useless residue; the refuse or offscourings. 1531 Elyot Gov. i. xiv, They..neuer tasted other but the fecis or dragges of the sayd noble doctrines. 1546 Supplic. Poore Com. (E.E.T.S.) 65 Symple creatures..taken for the dregges of the worlde. 1581 J. Bell Haddon's Answ. Osor. 358 Traditions of men: Mounckish vowes..pilgrimages, and innumerable such dredge. 1675 Traherne Chr. Ethics ix. 121 Matter is the dreg of nature, and dead without power. 1689 Hickeringill Wks. (1716) II. 495 For us who live in the Dregs of Romulus [cf. L. in Romuli fæce]. 1719 Young Revenge ii. i, Some dregs of ancient night not quite purg'd off. 1761 Hume Hist. Eng. III. lxi. 320 Low mechanics..the very dregs of the fanatics. 1876 C. M. Davies Unorth. Lond. 66 The very dregs of the population.
Turning to 'Scum' the OED is even more enthused. I will spare the reader and provide the link: http://findwords.info/term/scum.
Interestingly both terms relate to brewing and brewing by-products. It makes perfect sense that both brewing and dairying (and processing milk products) should provide a rich source of metaphors in our languages, given that these activities have very deep roots in our civilization. It might be an interesting further question as to why 'scum' seems (at least to me) to carry a greater sense of invective than 'dregs', although I might speculate that the association of brewing with 'dregs' lends the word some small skerrick of dignity.