While the OP specifies the US, some of the other answers are based on other countries. I'll respond mainly regarding the US, but I'll start with some comments regarding other countries. In many countries, *college* means secondary school, following on from primary school. It would never mean that to the average American.

Technically, a US college is a venue for obtaining post-high school (post-secondary) qualifications. It could be a 2-year degree, called an Associates degree, which may or may not be vocationally-based.  You can get an AA or AS in welding, math, Spanish, etc. These colleges are usually known as *community colleges.*

The other type of college grants Bachelor's degrees (not to be confused with the bachillerato granted in many countries to what in the US would be called high school students, who are typically going on to what in the US would be called college). There are 4-year liberal arts colleges, and there are subject-oriented colleges (which are now rare).

In the US, a university would be composed of multiple colleges, including one or more bachelors colleges, as well as a grad school. You really have to offer a Masters degree to be able to call it a university. 

Most Universities also offer Ph.D.s and possibly other advanced degrees (in the American academic world, we now call the MA and beyond *terminal degrees*), such as MD, JD, THD etc. 

Some places, like Boston College, have chosen to keep *College* in their name, even though they are universities.

I would not say "I'm going to university"; I'd say "I'm in college" (though for my last degree I was actually taking classes in three colleges of the university).  
I could say "I'm at the University of California", or "I go to Yale University", but not "I'm in university" (or "I'm in Uni") without it sounding a bit affected.

So if all I said was that my daughter was off to college, you wouldn't know if she was attending a small liberal arts school of 300 or a prestigious university, or even a local community college (but then why say she was "off to?") -- but you would know that she was studying at a level beyond high school.

People who are not in academia might use the term without knowing all these details.