Is age still a big issue even if she is legal?

I randomly walked in a graduate college party last week. I met many men and I was talking to one for about half an hour. We both were sarcastic, funny and we had this great connection. Then I asked him how old does he think I was, and he said 24. When I told him I was 18, he couldn’t believe me, and I showed him my ID. He suddenly seemed disturbed and after awhile, he looked down, touched my lap slightly once and walked away. I was shocked. He was only 27.

I did not understand… Is it really a big deal?