If it's a single building, as you say, then the total number of stories doesn't change.  It may be, for example, five stories on one side and seven stories on another, but we would say it's a seven story building.  Part of the structure being fewer stories doesn't decrease the total number of stories the building has.  Willis Tower in Chicago is 110 stories.  The fact that it has a sections that are only 50 stories, 66 stories, and 90 stories, doesn't diminish its count of 110 stories or lead anyone to call it anything other than a 110-story building.

That said, if you wish to clarify that there are sections that have fewer floors, then you simply say that.

(By the way, I'm American, so I'm using the American spelling "story" rather than the British spelling "storey," along with the American plural spelling "stories" rather than the British plural spelling "storeys.")