He dislikes me smoking weed?

my fiancee had a bad run in with weed when he was younger.. and now he can't smoke or catch contact. but i still smoke.. because it mellows me out.. now see i dont smoke all the time.. maybe like 3 times a week. but when im feeling a bit stressed.. i smoke.. and he has this huge problem with it.. i mean he lets me every now and then.. but he complains.. and he says it justifies him watching 24/7 now matter how often we have .. (i told him i hate him watching ) am i wrong? because i comprised.. i used to smoke daily.. now only 3 times a week. but he watches when ever he feels like.. what should i so?