Is there any pronunciation difference between both? Shouldn't IPA use one symbol per phonem/allophone? Curiously, this happens with the schwa, but not with "true" vowels, eg the A in car [kɑɹ].
Most Americans do not pronounce [əɹ] with two different, consecutive sounds (phones), but with one phone. You say "Shouldn't IPA use one symbol for phoneme?" The problem is that [əɹ] is two symbols for one phoneme. So [ɚ] is used instead.
For the other r-colored vowels, many Americans use single phones for [ɑ˞] and [ɔ˞], but many also use two, so [aɹ] and [ɔɹ] are considered to be adequate IPA representations of these. Wikipedia says:
In words such as start, many speakers have r-coloring only in the coda of the vowel, rather than as a simultaneous articulation modifying the whole duration. This can be represented in IPA by using a succession of two symbols such as [ɑɚ] or [ɑɹ], rather than the unitary symbol [ɑ˞].
For the other three r-colored vowels, [ɛɹ], [ɪɹ], and [ʊɹ], even fewer use a single phone than for [ɑɹ] and [ɔr]. (In fact, I don't know whether any Americans use a single phone for [ɛɹ] and [ɪɹ].)
If you try to represent American and British speech with the same IPA symbols, you run into big problems. What vowel do you use for palm, box, cloth, caught? Those are two vowels in American and three vowels in British speech, and there's not any easy way to deduce the American pronunciation from the British, and vice versa. So why not use [ɚ]?
And the American pronunciation of wanderer, /ˈwɑn.dɚ.ɚ/, is indeed different from the British pronunciation, /ˈwɒn.dər.ə/, in the der syllable.