I always had the impression that a plant was bigger than a factory and that a plant might contain several factories, but we wouldn't say that a factory contains several plants.
According to oxforddictionaries.com,
plant: a place where an industrial or manufacturing process takes place
the company has 30 plants in Mexico
factory: a building or group of buildings where goods are manufactured or assembled chiefly by machine
Not very specific.
Disregarding adjectives like "small plant" and "large factory," just looking at the words plant and factory, is one bigger than the other to the average native English speaker?