If you can't live WITH him but don't want to be with him, why don't you consider moving closer to where he's going to be, by yourself? Unless its financially impossible of course, but I really think thats your only option unless you can get used to seeing him only a few times a year.