When someone in the US says "When I was in college..." he can mean "college" but he can also mean "university", so I've been told. If that's true, how can we know which one he is talking about? If I say my daughter is off to college, can it really mean either one?
While the OP specifies the US, some of the other answers are based on other countries. I'll respond mainly regarding the US, but I'll start with some comments regarding other countries. In many countries, college means secondary school, following on from primary school. It would never mean that to the average American.
Technically, a US college is a venue for obtaining post-high school (post-secondary) qualifications. It could be a 2-year degree, called an Associates degree, which may or may not be vocationally-based. You can get an AA or AS in welding, math, Spanish, etc. These colleges are usually known as community colleges.
The other type of college grants Bachelor's degrees (not to be confused with the bachillerato granted in many countries to what in the US would be called high school students, who are typically going on to what in the US would be called college). There are 4-year liberal arts colleges, and there are subject-oriented colleges (which are now rare).
In the US, a university would be composed of multiple colleges, including one or more bachelors colleges, as well as a grad school. You really have to offer a Masters degree to be able to call it a university.
Most Universities also offer Ph.D.s and possibly other advanced degrees (in the American academic world, we now call the MA and beyond terminal degrees), such as MD, JD, THD etc.
Some places, like Boston College, have chosen to keep College in their name, even though they are universities.
I would not say "I'm going to university"; I'd say "I'm in college" (though for my last degree I was actually taking classes in three colleges of the university).
So if all I said was that my daughter was off to college, you wouldn't know if she was attending a small liberal arts school of 300 or a prestigious university, or even a local community college (but then why say she was "off to?") -- but you would know that she was studying at a level beyond high school.
People who are not in academia might use the term without knowing all these details.
In the US, there is little difference between a college and a university in common speech. Generally, institutes of higher learning with smaller student bodies call themselves colleges, while larger institutions call themselves universities, but the distinction is largely cosmetic. So you usually wouldn't know for sure, but that doesn't really matter.
In speech or writing, you would use college as the object of a preposition (e.g., "going to college") no matter whether the institution calls itself a college or a university; "going to university" would be understood, but sounds affected when spoken by a natural born American.
From a technical perspective, a university is a school of higher education that offers a graduate program, while a college does not. There are exceptions to this rule, but it mostly works.
In practice, though, these are seen as VERY SIMILAR academically in the US. In terms of my education, I can get a good education at a program that only offers a 4-year bachelors and I can also get a good education at a University that offers a post-graduate degree. I get the same education at either type of institution. That said, universities are seen as more prestigious because post-grad work creates the opportunities for research and in general, they will offer more programs and more variety of programs. That's why Americans don't see it as important to differentiate between which of the two schools they attended.
To answer your question, you cannot tell if someone went to a university or college based on a statement like "I'm in college" or "I went to college." However, if someone is talking about post-graduate work, they will say something like "In my post-grad..." or "For my masters..." In general, there's really no reason to care whether they went to a college or a university.
I feel compelled to add a non-US answer, as these words have quite distinct meanings in the UK.
The US-centric use of the word "College" as a direct alternative to "University" can cause considerable confusion to older UK English speakers, and indeed can feel annoyingly vague even to people who are aware of the different meaning across the Atlantic.
The common distinction made is that universities offer graduate programs and colleges do not, but that is not entirely true. Many colleges do have graduate programs, but they are typically applied masters programs with little to no research components. The true distinction of a university is that universities have research programs and PhD programs. In fact, universities tend to put a lot of emphasis on research.
Due to the emphasis on research, it is not uncommon for classes at universities to be taught by TAs and grad students, this is especially true of entry-level classes. The larger the university, the more likely the lower-level classes will be taught by aids and students. At smaller colleges, however, the classes are generally taught by the actual professors.
As with most things, there are exceptions. There are smaller universities with less research emphasis and there are larger colleges that do have research components.
It is also useful to note that a college can be a division of a university. For instance, you could study under the College of Arts and Sciences at Cornell University. And, to further confuse things, sometimes these divisions are called "schools," such as the NYU Tisch School of Arts.
All that said, in the US it isn't pejorative to say that someone went to "college."
I have never heard an American refer to "university." They say "college" just about exclusively. This is regardless of whether the place they got their bachelors degree was actually a university or a college (also, as others have mentioned, sometimes "universities" are made up of smaller "colleges"). Literally the only times I've heard Americans say something like "when I was at University" is when they are talking to a foreigner. It's always "when I was in college" or sometimes, "when I was in school" (if it's clear from the context that they are talking about college).
Likewise, nobody would ever say "university" to mean any kind of post-bachelors degree. They'd say Med School, Business School, Grad School, etc.
In many large American universities the "University" is the high-level organization and the "College" is the specific subordinate organization that focuses around a more specific area of study. It is common to see "University A, college of B". In common speech "Going to college" is what we say, but what we usually mean is "going to university".
The semantics are blurred today, mostly for reasons that have a political history behind them, but you can still see the original meanings in use at most American institutions older than 80 years or so.
Various headers read "Texas A&M University, College of Engineering" or just "College of Engineering", because that is a specific college within the Texas A&M structure. Within that college they have further subdivisions by "Department" (electrical engineering, computer science, etc.).
Some universities define a "college" as a specific campus location and focus there on a specific area of study. Some universities even call themselves a "university system" and each campus a "school" and each department a "college". The overall intent is that these terms imply a hierarchy, whether they are commonly used in that sense or not when speaking casually (and its worth noting that American nearly always speak casually, even on formal occasions).
A college is a school that you go to for learning a skill for a job as an adult. For example, "I went to college to become an Engineer."
A University is a collection of colleges that have grouped together. If you look at a University website they will often ask, "What school would you like to apply for?" Here 'School' and 'College' are similar. They may say the 'College of Engineering' for example.
The idea of a University is to create the 'Universal Man'. Someone who is well educated in many areas of life. So when someone goes to University they are learning many things. If someone says 'College' they may mean something more specific, but a college most often is with other colleges making up a University.
Community Colleges are a bit of a stepping stone to a University as they may not have a large enough group of colleges to be officially called a University.
If you are talking with someone you cannot really be wrong by asking, "What college are you at?" That is a good open ended question to make good conversation, because they could answer with different kinds of answers.