Didn't the Vortigaunts basically chase him off at the beginning of HL2E1? Now, call me an easily charmed fool, but I liked the Vortigaunts. They were friendly, personable and genuinely seemed to be on our side. I mean, they got their asses conquered by the Combine and were used as shock troops against us during the initial invasion, but quickly began defecting once it was clear that humanity might actually be a legitimate threat to the Combine's power.
Either they drove G-Man off because they didn't want him yeeting Freeman into stasis for another 20 years, or they're legitimately afraid of the fucker. I mean, remember what he was saying about "employment opportunities" at the end of HL2? Creepy shit, and very ominous. Something tells me that fucker does NOT have our best interests at heart. Fuck, even Breen was aware of him pulling Freeman's strings.
I trust the Vorties a lot more than I trust some dimension hopping suit with a speech pattern irregularity.