First of all, there are other meanings for the word brother outside the family context
2 : one related to another by common ties or interests
3 : a fellow member — used as a title for ministers in some evangelical denominations
4 : one of a type similar to another
5 a : kinsman
5 b : one who shares with another a common national or racial origin; especially : soul brother
The etymonline states that: "As a familiar term of address from one man to another, it is attested from 1912 in U.S. slang; the specific use among African Americans is recorded from 1973."
So, you should be aware that even if you are speaking specifically of the meaning of the word in subcultural context of African Americans, the word itself still carries the sense of kinsman, fellow member or simply someone who is related by common interest or ties.
Now, regarding who should use it for whom: another piece of the puzzle is that the term brotha is Jamaican English; which though it should not be confused with Rasta movement, often is (do notice that the context is not race anymore).
Now you have at least two subcultural contexts (race, religion), however for the true appreciation of applicability you should consider still more contexts, for example use in gang slang or in urban slang. Illustration of various contexts can be seen on urban dictionary for the term brotha.
For your questions:
- what effect is conveyed when non African Americans use it when talking to one another: See urbandictionary.
- if African Americans would call a non African American brother: Yes. (but keep in mind possibility of myriad of different contexts, e.g. A middle class African American addressing a declared racist would definitively not use it, except sarcastically i.e. at least one of the before mentioned meanings must be satisfied)
- how African Americans would react if called brother by a non African American: Depends mostly on perceived sincerity. Do note that if the slang is not used properly that it will be most likely perceived as insincere.
- how non African Americans would react if called brother by an African American: Again it depends if it is justified.
All in all, the use of the word must be justified in some sense of it. Take note, when you say that you 'gathered that it was OK for African Americans to use the word when talking to one another' - that this is a statement that is a bit too general. Is it ok for every person in London to call each other 'old chump'? Is it appropriate? The answer is that it depends on the social connection that is shared and the context.
Similar situation is with word brother and the variations of it bro and brotha.