Well, he had an affirmative duty to his wife to be faithful, that's one thing. It's assumed that he could have brought pressure to softly force his "partners" into it, which may not be far from the truth in some cases (though surely not all).
There's also the smarmy "But my dear, the Patriarchy conditioned me to do it" attitude. Nothing says "insert chainsaw here" like blaming society for one's voluntary assholery.
Truth be told, I'm glad Ms. Cole is mad at him instead of blaming his mistresses for stealin' her ma-yan. Too many women go the latter route and let the ma-yan off scot-free. I'd like to see the mistresses get what's coming to them as well, but I'll cheerfully wait on that just to see something that doesn't happen nearly as often as it should.
