Any word used to describe a bad thing necessarily becomes a slur.
We used to call handicap people "lame", then it became a slur instead of a description, so we shifted to "retarded" for politically correct reasons (someone being a bit tardy compared to others), and now it's a slur. The cycle won't stop because it cannot stop.
Same thing with "nigger" that became "colored people" that became "black". Since the thing is bad, a mere word for a color is bad, so now you're supposed to use "African-American" or some retarded shit.
Yes, if jews are bad in the collective unconscious, calling someone a jew is a bad thing. No matter if he's a jew or not. That's why you're supposed to use a softer word like "jewish" or be a good goy and never mention to begin with.
Notice that "white" has never been a slur.