I think you give those people too much credit if you let this bother you.

No, I don't think you are a racist. A cocky opinionated little B maybe, but not a racist.