There are [many answers][1] for this, but looking at the dictionary we get:

> [at][2]: In or near the area occupied by; in or near the location of
>
> [in][3]: Within the limits, bounds, or area of


People are usually using *in* to note a general location and *at*  for a more specific location.

> I'm in the building, at the front desk 
>
> I'm in New York, at the conference
>
> I'm in New York, at the Empire State Building

EDIT:
But note also the difference when *in* is used to indicate *inside*

> I'm in the elevator = I'm inside the elevator
>
> I'm at the elevator = I'm near the elevator




  [1]: https://www.google.com/search?oq=differences%20between%20%22in%22%20and%20%22at%22
  [2]: http://www.thefreedictionary.com/at
  [3]: http://www.thefreedictionary.com/in