The infidelity, and the 8 year long engagement to his fiance, are what made me think Vic did do something inappropriate at some point. In fact, I still believe there's too much smoke so there has to be a small fire somewhere in his past.
I'm pretty sure you can throw out the garbage about pedophilia, rape, sexual assault, harassment, homophobia, and anti-Semitism. A fan being uncomfortable about a hug/kiss is just going to happen; meet 10,000 fans, have 1% of meetings go wrong, and you get your "100 women" just as a margin of error.
He's admitted to infidelity, propositioning the twins, and pulling Marchi's hair, none of which are illegal/immoral, but they suggest the kind of incautious attitude that gets you that one legitimate skeleton in your closet. Like propositioning a girl who you thought was legal but was actually underage, or misreading a signal and grabbing someone's ass. I don't think he's done anything remotely near as bad as what he's been accused of, but you could probably unearth something he's legitimately sorry for.
The problem is his accusers are such exceptional liars, backstabbers, and dumbasses that I can't tell which accusation might be legitimate. The outrage mob amplifies every minute accusation, making them all sound suspect simply by their inclusion. ProTip: this is why you don't lie about sexual abuse, you idiots. This is why you don't listen and believe. Because it fucks up the reception for legitimate victims and the people trying to help them.
(ManJaw, I'm looking at you. You had the most likely reported/traceable accusations of the group, then spent the last few months lying so hard and so frequently I had to stop mocking Brianna Wu just to have time to process all your insane shit. The fall in credibility is amazing to behold.)