I will study comedy at XYZ theater
Sounds more intuitive.
Which one is grammatically correct and why?
It depends on how you perceive the XXX Theater. If you're talking about the building, occupied by the XXX Theater, then you should use in - I will study comedy in the [building of] XXX Theater. Same as I will study comedy inside the XXX Theater
If you're talking about XXX Theater as a whole, then at would indeed be correct. I will study comedy at the XXX Theater.
"At" is more correct
The same reason why we don't say: "Have dinner in my house.", but rather "Have dinner at my house."
The reason "at" is used is because "in" refers specifically to within the building, whereas "at" refers to the building and its vicinity.
That's why we say " I'm at the station in New York."
You are in the vicinity of the station, but you are within New York.
So you study at XXX theatre, as well as its vicinity, i.e. it's smaller buildings and offices surrounding it, but you aren't necessarily studying only when you are inside the theatre itself.