Either question him, or break up with him. Even if it pisses him off, it still may give you peace of mind, which you need. However, notice that I said it may give you peace of mind. Knowing what they did or how he felt about it may just make you feel even worse about this.