When is the right time to say 'I love you'?

Me and my boyfriend have now been going out for 1 week and 1 day :) We are really close and talk all of the time. Last night I was talking to him on the phone and he said I Love You for the first time, I was a bit freaked out and didnt know how to respond. I ended up saying I love you too. Is this too early to being say it seeing we are only 15?