One of my letters of recommendation is being written by a professor of my college, who is also the dean of research. However, he has never taught me, I've only done a project under him.
So what is our "relationship"? Is it be 'Professor' or 'Project Mentor/Guide' or something else?
It depends on the context of your letter and the intent.
For example, if he was the leader of a project you worked on and you did not work directly under him. I would recommend the use of 'facilitate' in this instance and also 'Project Lead'
Example:
I was involved as [What was your title] doing [What did you do] under [Professor X] who was the Project Lead and facilitated my development.
The reason is "mentor" usually has a personal context to it. Meaning that you were literally being taught, led, advised, etc., by the person in question. They took a vested interest from a personal capacity in assisting your growth. From your response, this wasn't the case. It could give the incorrect impression as a result.
Facilitate is different from teach/tutor in this context because it means he helped you help yourself. As a Project Lead, that is usually what they do. They help you do your own work and improve your skillset to expedite the project.
The combination of these two (facilitate, Project Lead) would describe your relationship in the way you are trying to describe.
On a side note: not Project Manager, which again has a personal touch to it (which essentially means he's your mentor).
This would be well-described as a mentorship, the professor being the mentor, and you being the mentee. The word mentor is less associated with teaching, and more associated with guiding, which seems to fit your situation perfectly.
