How do you 'let someone down' or tell them in the nicest possible way that you are not interested?

‘Just be honest’ is never that easy.
Im in a situation that i bought upon myself. terrible really.
I kissed a boy a few impressionable years younger than me (on two occassions) whilst under the influence and now he has asked me out.
Im sure he is a lovely person, but he is also friends with someone else i used to date and i just dont want to go there. I dont even think he is aware i used to date his friend and its really not something im willing to bring to light.
i now have this icky feeling inside.