There are many answers for this, but looking at the dictionary we get:
at: In or near the area occupied by; in or near the location of
in: Within the limits, bounds, or area of
People are usually using in to note a general location and at for a more specific location.
I'm in the building, at the front desk
I'm in New York, at the conference
I'm in New York, at the Empire State Building
EDIT: But note also the difference when in is used to indicate inside
I'm in the elevator = I'm inside the elevator
I'm at the elevator = I'm near the elevator