Basically for the same reason that one blue sweater plus one blue sweater do not become two blues sweater.
Blue describes the head sweater, and whether there is one, or there are more, they are blue, not blues.
The same goes for, say, candy stores. Candy attributively modifies the head noun store, and we only pluralize the head noun.
Likewise, in attorney general, the head is attorney which is modified by general. So when we form the plural, we pluralize the head, not any attributives that modify it.
Again, in passerby, by modifies the head passer, and when pluralizing we pluralize the head.
What throws you off is that the head noun is not always preceded by its modifiers. Usually we add adjectives and the like before a head noun, but sometimes we don't. Even if we add them after the head, we still follow the same rules for the plural: pluralize the head noun, not its modifiers (unlike, say, French!)
Once an allocation becomes so common that we see it as one word (adding a hyphen, or even removing any space in between the constituent parts) a plural may be formed considering the expression one word, hence an -s at the end of attorney-generals is possible.
As for passerby, somehow it seems we create the plural as if we still wrote passer by. I guess passerbies just doesn't look good — and if you want to write passerbys, you are opening a whole new can of worms. Then the next question will be why that plural is different from babies, ladies and derbies!