Should I back off?

My husband and I have been under alot of stress with kids, working every day, money and spending time together. We have only been married a year, most weeks we only spend about 2 nights together if that, and we seem to fight most times when we are together. It seems as if we want different things when we are together, I like to be affectionate,like sex, to talk and interact, to hear nice things and he likes watching tv, piddling around the house and sleeping. Recently we got into a fight because I was upset he didnt even acknowledge it was valentines day, no happy valentines day, I love you, not even sex, he told me how ridiculous I was being and said I dont make him happy and he feels smothered, should I just back off and leave him alone? Or is backing off like giving up and saying you dont care?