It is really strange. It must be a combination of, DSPs tractors aren't as obsessed, he's much less known as a lolcow then we expect, he and Khet avoid the outside and if they have to go they keep their time spent in a public space as minimal as possible (mad dash from car to store and back).
I'd put my money on him being less of a known lolcow than we expect. Because this subforum is dedicated to him, it's easy to forget that, to the wider internet, Phil is at most "that guy". Whenever he gains some notoriety, people know who he is, but then express immediate surprise that he's actually still around. I think to most, they assume Phil's career died around the time he went to Twitch.
Even here, where he is popular enough to get a subforum, I think he's kinda niche. And I think for a lot of us, we only remember Phil when we login and see new posts and forget about him a few minutes after we logout.
But the hermit aspect is probably a factor as well. Phil only goes out once a week. For the detractors that try and make a living off of Phil, that's not enough to work with. Much like Phil, they need to constantly churn out the most low effort content possible, so there's no time to dedicate to some kind of gayops to get a photo of him waddling around in the wild. And even if they did, while it would probably get some laughs and go viral among the detractor community, there's no real way to monetize it.
Trying to do a gotcha style video would also have a similar effect and ultimately be a nothing burger. He'd just quietly shuffle away like a pussy, we know how he handles face to face confrontation. For dentractors, the real gold is clips of him ranting and raving on stream.