Is it my imagination or are these people racist?

Here’s the scenario, I moved to this very small rural village last month. My husband and I are the only black people here and since moving here only one person has spoken to us. Okay, not a problem, we assumed that maybe they were distrustful of outsiders, so we shook it off. Last night my husbands sister came to see the new house and loved the peaceful quiet area. She was on the deck when immediately the next door neighbor ran over and talked with her for over an hour. I didn’t get home from work until much later. Then this morning while I was hanging clothes on the line the same woman ignored me. Didn’t say a word. I go back in the house. Sister-in-law goes out to take a walk and again the next door neighbors are chatting her up! WTF!?!?!? Oh maybe I should have mentioned that my sister-in-law looks white. My husbands mother is half white and his dad is black so some of their kids came out brown (like hubby) and some came out almost white (like his sister). I knwo I shouldn’t care but I have to admit my feelings were hurt when she told me how these people chat her up every time she walks out of our house but in a month they have said nothing to us. The neighbor also confided in sis-in-law that she caught another neighbor was looking in our window because he was concerned when he heard that black people moved in. Funny enough though the guy caught peeping in our window is black too (so technically we are the 2nd black family here). Okay after this long story I guess my question is what are they afraid of? Should I make a bigger effort to put them at ease or just ignore them? Is it too naive of me to think that in this day and age people would judge me based on who I am and not my race? Feedback people, I feel so bad right now. PLEASE!