While the OP specifies the US, some of the other answers are based on other countries.  
I'll answer mainly to that, but first for some other countries.  
In many countries college means secondary school, after primary school. It would never mean that to the average American.

Technically (in the US), a college is for post-high school (post-secondary). It could be a 2-year degree, called an Associates degree, which may or may not be vocationally-based.  You can get an AA or AS in welding or in math or in Spanish or...  
Those colleges are usually called Community Colleges

The other type of college grants Bachelor's degrees (not to be confused with the bachillerato granted in many countries to what in the US would be called high school students, typically headed on to what in the US would be called college).  
There are 4-year liberal arts colleges, and there are subject-oriented colleges (rare now).

In the US, a University would be composed of multiple colleges, including the bachelors college (or several), as well as grad school. You really should offer a Masters degree to call it a University.  
Most Universities also offer PH.D.s and maybe other advanced degrees (in the American academic world, we now call the MA and beyond terminal degrees) such as MD, JD, THD, etc.  
Some places like Boston College have chosen to keep the name College even though it is a university.

No, I would not say I'm going to University, I'd say I'm in college (though my last degree I was actually taking classes in three colleges of the University).  
I could say that I'm at the University of California, or I go the Yale University, but not I'm in university (or Uni), without it sounding a bit affected.

So if all I said was that my daughter was off to college, you wouldn't know if she was in a small liberal arts school of 300 or attending a prestigious university, or even a local community college (but then why say she was "off to?") but you would know that she was beyond the high school level.

People who are not in academics might say something without knowing all these details.