As a white person, if a blatant racist comment is made against me, and if I speak out agaisnt it??

If i speak out against it, and say, that was not a nice thing to say, and that is a racist statement. Is it fair for people to attack me for defending myself or speaking out against the statement? I see a lot of that sort of mentality in america, that somehow the person that did nothing wrong and was blindsided by a racist comment is somehow the racist or playing the race card for speaking out against the person who started the mess in the first place.