So ... I gotta be "that guy" for a minute here. I don't mean this to sound provocative; I'm just genuinely curious.
You've known the guy for years, find him completely trustworthy, have never once had any kind of problem with him or seen any indications whatsoever that he's violent or has anger issues (either now or in the past), have never been hurt or threatened by him and felt comfortable with him enough to meet him alone in a safe but secluded place to discuss business. Yet you were so terrified of him (even if just for a moment) that it impacted your job performance and made you negotiate a worse deal out of fear of angering him if you pushed harder.
Why? Dude literally didn't do anything but you were still deathly afraid of him.
I guess I can understand if it was just some primal/instinctual "gut feeling" since we don't necessarily control those consciously, and I suppose it worked out okay for the guy since he apparently got a better deal than he probably expected and he didn't get #metoo'd, but this just seems weird to me. If it was just a general fear of anyone and everyone who's bigger than you, it would make more sense to me, but this sort of sounded like you were afraid of just this guy in particular.
I'm beginning to understand the MGTOW guys a bit now. You can be a trusted colleague with a good reputation, do literally nothing wrong and still end up the bad guy