Which is correct, "I worked on a project" or "I worked in a project?"
Should I say "when I was at the university I studied math" or "when I was in the university I studied math?"
There isn't going to be a quick way to match up propositions. There are some general rules that fit a majority of the cases but the exceptions abound.
Unfortunately, changing prepositions around will have drastic effects on its meaning. Instead of trying to catch and explain all cases, I have decided to simply use your examples and explain what they mean:
"On" is typical here and will be correct most of the time. Working on a project implies doing work related to the project and is similar to working on a house or working on a letter. You are performing an action (working) on an object (the project).
The tricky side of "on" comes with phrases like, "I worked on a farm." Working "on a farm" means doing work while located on a farm instead of working on the object of a farm.
A good test is to try replacing "worked on" with "completed": "I completed a project" or "I completed a house" sound fine while "I completed a farm" drastically changes its meaning.
Working "in" something is more akin to being located inside of the object. "I worked in a building." A "project" can certainly contain things and so you can technically work "in" a project.
Here, "at" is mostly referring to your physical location. You did work while being located at the university. This generally refers to students enrolled in the university but also applies to people who were just visiting or possibly working there in a way unrelated to learning (such as the janitor.)
Again, "in" here implies being "inside of" or "contained by" the university. To say, "I am in university" generally means to be enrolled in one of the university's programs. Unfortunately, it can also mean being physically inside of the university buildings... so I am not helping much.
The most common case is to say, "When I was at..." but there is a mix of things happening here. People will ask you "Where do you go to school?" The importance seems to fall on the physical location but this usage does not cover strange exceptions such as enrolling in classes over the internet where you never move from your chair but are considered "at" a particular university.
In any case, I hope this helped somewhat.
I worked on a project. (In sees very occasional use here, but is much less common and sounds strange to me.)
When I was at the university I studied math. (In is a bit more common in this case, but still less often seen. However, one would say "When I was in college I studied math".)
"I worked on a project" is correct. In would be wrong in this context. One can work in a particular place or location (obviously), as well as a field of endeavor, a professional or academic department, or a certain artistic medium:
If you want to say you studied math while a student at a certain university, then you should use at. Thus, your correct option would be:
In the university would only be correct when referring to the physical space that makes up the institution, e.g.
Even when used correctly, in the university is still quite rare. In place of the above example, one might hear instead: