Would you trust your access info to an online financial site?

My husband thinks it would be a good idea to use an online program such as mint.com to track our finances. I am not comfortable with letting an external website have my login and password to my banking & other financial websites. A large part of my concern is internal site security & outside hackers. Any opinions to share?