My brother is distancing himself from me... What's going on?

Alright, well I’m 18, and I have an older brother, Stephen, who’s 20. We adopted Stephen when he was 7; I was 5. It’s very clear, too, since he’s a rather tall, muscular black guy, and we’re all rather short and very white. I’m 5’2’’ and I’m about as pale as you can get. Haha Anyways, I love him very much. He’s the best brother. He’s always been there for me; he’s very protective of me. I used to be annoyed by it, because every guy I dated would have to pass his inspection (lol), and he would constantly have an eye on what I was doing, but I don’t mind now. I know he just cares about me. He’s just a great brother. He’s so much fun, and he’s just the kindest person.

Anyways, because we look so different, everyone has always given us a lot of crap about being siblings. People in my graduating class would make fun of me for “the big, muscle-y black guy” being my brother, and he always got all kinds of b.s. from the people in his class about the “tiny pale white girl” being his younger sister. And because we spent a lot of time together in school, seeing as how we’ve always gotten along fairly well, people would see us together all the time. This just fueled their fire, and we always knew when we were being talked about. It didn’t used to bother either of us, and I certainly never cared, and still don’t, but during Stephen’s senior year of high school (my junior year), he started acting weird about it. He started distancing himself from me at school a bit (not in a way that was hurtful to me. He just sort of drifted from me), but at home, he was exactly the same as always; fun, joking around with me, hanging out with me, etc. It still bothered me that he actually cared what they thought, though.

Anyways, we both go to the same community college now, and Stephen has sort of gotten together a new group of friends. I’m not particularly fond of any of them. They’re not terrible, but they’re different than him; they can be somewhat obnoxious and seem like trouble, honestly. Stephen and I are both really good kids; we don’t drink, smoke, party, or any of that. But whatever, I’ve been trying to give them the benefit of the doubt. However, they recently came over our house, and met all of us (myself, our parents, and our 15 year old sister Breanna). They reacted the exact same way everyone in high school did. I heard them giving him a lot of garbage about looking so different from all of us. He invited me into the living room when they were over to hang out with them, and they seemed really judgmental the entire time. Since then, Stephen has seemed really distant, and sort of… quiet. He seems upset all the time, and I try to go in his room and hang out and ask him if he’s alright and he just says something like he “has a headache” or something. What’s going on? What do I do? I miss the normal Stephen, and I’m worried I’m losing my brother. He’s my best friend. :( What do I do?