The common distinction made is that universities offer graduate programs and colleges do not, but that is not entirely true. Many colleges do have graduate programs, but they are typically applied masters programs with little to no research components. The true distinction of a university is that universities have research programs and PHd programs. In fact, universities tend to put a lot of emphasis on research.
Due to the emphasis on research, it is not uncommon for classes at universities to be taught by TAs and grad students, this is especially true of entry-level classes. The larger the university, the more likely the lower-level classes will be taught by aids and students. At smaller colleges, however, the classes are generally taught by the actual professors.
As with most things, there are exceptions. There are smaller universities will less research emphasis and there are larger colleges that do have research components.
It is also useful to note that a college can be a division of a university. For instance, you could study under the College of Arts and Sciences at Cornell University. And, to further confuse things, sometimes these divisions are called "schools," such as the NYU Tisch School of Arts.
All that said, in the US it isn't pejorative to say that someone went to "college."