Not being attracted to a black man who contacts you is not racist.
Saying you're not attracted to black men clearly indicates one of two things:
(1) You think all black men look so similar and so alike that you can automatically know that you won't be attracted to them just by knowing their race or,
(2) You aren't attracted to them because of their skin color.
So, you're not attracted to them because you think they all look alike or you don't like black skin. Either way, that's racist.
More information about text formats