His male pattern baldness is what does a lot of it IMO. He has bad skin like CWC though, always sporting a 5 o'clock shadow somehow as well. I know he has some weird thing with his upper lip showing too much teeth, but it also looked like he has had receding gums for a while as well, which make his teeth look bigger, he would probably bleed profusely if he ever brushed his teeth, I have to imagine he has a really bad case of gingivitis.
It's funny that he manages to look so haggard and worn out despite being obese, because the one thing fat people usually have in their favor is a younger look because their bone structure isn't visible and the fat tends to puff out and hide wrinkles and other skin related signs of aging, but Phil doesn't even get that benefit. It is all rounded out by his potato features. He is truly a specimen to behold.
At a glance, though, it comes back to his hair loss, not only is his hairline significantly receded, but the hair he does have remaining is extremely thinned out and scraggly. He is at the point where normal men just say fuck it and go for the Stone Cold Steve Austin look, but he is awkwardly clinging to his hair and when he is seen without a hat on he basically looks like the 1980s Richard Simmons.
And it's just fantastic that on top of everything bad about his looks, he will do shit like shave his eyebrows, dye his stringy unwashed hair bright pink and put on lipstick. I swear to fuck he is basically one of The Joker's henchmen.