The offensiveness inherent in the word gaijin is that it is essentially lumping together all other countries' peoples. How about a personal anecdote?
I was once in the presence of very diverse company for a work function where one Japanese person made several comments about us that were quite broad sweeping generalizations. Nothing particularly rude in and of itself, but the group she was refering to included myself (a Canadian), an American, a Spaniard, and a Khazakh.
Although you might argue that Canadians and Americans are somewhat similar, it felt pretty weird to be put in the same box as people from Spain and Kazakhstan.
In this way, the word gaikokujin is no less offensive really, it just signifies that they are attempting to be polite. However, courtesy and ignorance are not mutually exclusive. The fact that this word is frequently used points to elements of Japanese culture that are uncomfortably nationalistic. It often makes one feel that Japanese believe that are uniquely special, that the world outside of Japan is totally different or incompatible with Japanese values, that Japanese people are biologically very unique even, and ultimately it's that although Japanese people don't necessarily think they are better than everyone else, they do think that everyone else is worse than they are.
Getting back to the word gaijin, I think it's as simple as translating the word. I think the word foreigner is just as problematic, for the same reasons I mentioned above. That's why you rarely hear English speakers refer to someone as a foreigner. We at the very least take the time to categorize people roughly into european, asian, latino, and black. Of course, these are obviously problematic labels as well.