In the grand scheme of things what you propose is to ignore one grain of sand on a beach. Besides the sheer futility of this move the question would then become what makes this grain of sand any more significant than all the others?

It is from the aspect that it's very offensive if you aren't friends with whom you use it on, andf even then how and when you use it. Only the racial aspects are missing from it, everything else applies.

Just as the N word is not offensive when used among close friends.