He doesn't physically abuse you, does he???
Has he EVER hit you? 'Cause if he has, then he WILL do it again and again...and the only remedy to that is to kick him to the curb.