The key is the sequence of events, and the point to remember is the fact that being someone's brother does not change over time (apart from some specific circumstances).
Normally the second sentence would be correct - the events have happened in the past, and therefore you would use:
If he had been my brother, I would have helped him
However, if he had been your brother he would still be your brother now. Therefore, using the first form is still OK:
If he were my brother (now, and presumably at the time of the events too), I would have helped him
It wouldn't work, for instance, where the events are limited to a timeframe:
If he had been thirsty I would have given him some water
You can easily see that him being thirsty now would have no bearing on giving him water in the past.