Is it wrong to stay in a friendship when you know the other person has feelings for you and you don't for them?

I’ve had this good friend for about 8 years now. We have a lot of fun together, share mutual friends, and have a lot in common (political views, love great food, wine, talk about travel and other cultures, etc.). He is 20 years older than me and I’m not attracted to him physically. BUT he is to me. He has never made a move on me but it is clear how he feels. Sometimes I wonder if it is wrong to spend a lot of time with him, knowing he would want more in a heartbeat if I felt the same. Any thoughts? Thanks!