You need to leave. Sorry, but if he says he forgives you but continues to drag you through the muck, your relationship will never recover. He doesn't know how to let it go and is being vindictive. Based on the nature of your crime, sounds to me his behaviour has been as bad, or worse than whatever you did. No wonder you felt disrespected and went looking for affection elsewhere.
Some people use situations like this to their advantage and as a point of control. That's not fair. What you did was wrong, yes. You've done your best to apologize and seem to have learned your lesson. The fact you made a mistake doesn't mean he gets to hold it over you the rest of your relationship.
Second thoughts can generally be amended with judicious action; injudicious actions can seldom be recovered with second thoughts.
--Cyteen by C.J.Cherryh