The OED lists only one non-metaphorical sense of the word caravanserai - (from caravan - etymology Persian).
A kind of inn in Eastern countries where caravans put up, being a large quadrangular building with a spacious court in the middle.
This is endorsed by Wikipedia which provides much the same meaning.
However I have most often heard caravanserai used to describe a motley transportantion on a number of vehicles - such as of a circus moving from one place to the next. Of course a caravan was originally: A company of merchants, pilgrims, or others, in the East or northern Africa, travelling together for the sake of security, esp. through the desert. So the notion of transportation was inherent to the word's beginnings. Hence I am puzzled by the idea of caravanserai being a fixed building or settlement.
Consider the following use - which accords entirely with the way I have most often heard the word used in Britain- from Six Wives: the Queens of Henry VIII by David Starkey (London 2003) p.230.
It was in any case a day of upheaval at Court as it was the beginning of the Progress. The whole courtly apparatus of 'portable magnificence' - the tapestries and cushions, jewels and plate, household utensils and the King's own clothes, bedding and travelling library, medicine chest and personal petty-cash - had been packed into their special bags, boxes and chests and loaded on to carts. The carts had been covered with bear hides to protect them against the elements and the great caravanserai of the Court stood ready to depart from Greenwich to the first port of call of the Progress: Waltham Abbey in Essex.
This seems to refer, as I would have expected, to the travelling body rather than to anything fixed.