"Contingent" refers to an employee who can either be full or part-time, but who is not an official employee of the company they work for. The more casual term for this is a "contract employee" or "contractor". It generally implies that you are only with the company temporarily for a short term project. I don't think this would qualify as casual work since contractors tend to have highly specialized skills.
There's another term called "Temping" that is a bit closer to casual labor. It's when someone is employed at a "Temp Agency", which is a company that selects people to work temporary contracts at various companies. This can be almost identical to contracting or almost identical to casual labor. It all depends on the specifics. Sometimes people will specify that they are temping full-time or part-time.
Another possibility is "Working Odd Jobs". This can overlap with temping, but not contingency/contracting. This could be the closest approximation to casual labor we have in American English. It implies that the person working odd jobs is taking short-term, part-time gigs only, generally have in unskilled labor. It's common for world travelers to work odd jobs while traveling, since that type of work is easy to pick up and put down.
The last term I can think of is "Working Part-time". Part-time workers have fewer hours and more flexible schedules. They tend to be working part-time at only one job for an extended period of time. This can qualify as casual labor, since this would be the mother who works at the local store a few days a week, or the student who works here and there while attending classes.