He's on a drug infinitely worse though, his massively over-inflated ego. What we're seeing is what is (or at least should be) expected of anyone who spends a long enough time surrounded by sycophants and yes-men suddenly having criticism that they cannot ignore, a full-on nuclear spergout.
I don't think Jim was the first to get under his skin though, I think Spencer telling him "you aren't as smart as you think you are, and that's a terrible place for any man to be" is what rattled him so hard he broke. As far as why it got to him so badly, while it was a line delivered to him during a "debate" with an ideological opponent that he regarded as beneath him intellectually, in front of thousands of people live, I don't think that's why it fucked with him so badly. I think he never recovered from that afterward because of all that and the fact that somewhere in his head he knows that it's true, and the part of him that does is clawing around in his brain like a rat under a hot bucket.