Indeed it is used that way. It's a way of SJWs trying to bluff to get their opponents to back down. They accuse people of bigotry and have them try to defend themselves against the ad hominem of it and avoid the blowback of being called a bigot.
This is a good point, and it's basically the definition of an ad hominem argument. It is also, in many cases, a very convincing argument, and logical fallacies basically exist as arguments because they do work. For instance, if someone is literally a Nazi, for instance, you generally do not really care what they think or say.
Logical fallacies are used as a shortcut to thought, and this isn't always bad. Some sources of information (such as literal Nazis) are rarely sources of good information, and just presumptively discarding their output is efficient. Google "rational ignorance" for other times this shortcut to thought makes sense. It simply isn't worth it to pick through turds in the hope of finding a kernel of corn.
It's easily abused as a tool of "debate," though. For instance, it is nearly inevitable that anyone who criticizes anything the state of Israel does will be accused of being anti-Semitic or the old stand-by, a "self-hating Jew." The vast misuse of this tactic has robbed it of a lot of its sting as a result, though. And of course if you're on the other side, you're automatically a terrorist supporter.
SJWs have taken this to a new level, though. Literally everything they don't like is racist. Or sexist. Or some word they just made the fuck up.
Part of the reason the racism allegation is so appealing is that people almost automatically will feel compelled to defend themselves against it. Seriously, in today's society, being a racist is basically being a complete piece of shit, and the very fact that this is the case is, as you point out, pretty strong evidence that seriously, racism is something we're trying to eradicate as a society.
However, this natural desire to defend yourself against this accusation is what whoever hurls it wants. It is basically sand in the face. I think at least when the accusation is entirely baseless, it should just be disregarded and you should keep on whatever argument you were making in the first place.
It is basically an irrelevant and ridiculous slur, and should be treated as such.
Another possible response is just to hurl back slurs and insults of your own, and I like this one, but find it hard to recommend it.