I don't have much doubt that the dude was a real census worker. The root problem was he was very dumb. When I finally told him to get the fuck off my property he replied "I'm not on your property". That was in my yard, less than 5 feet from my front door. That made it clear that we'd left the path of reason, so I gave myself permission to lay hands on him, and now I'm being charged with assault.
I think census workers are granted special indulgences, like utility workers, so "get off my property" may not work for me.
To get back on topic. If he'd been a big black dude, instead of an old retard, I probably wouldn't have wanted to risk getting my ask kicked. Either I would have gotten a gun, or called the cops to come shoot him. Either way, it wouldn't have gone as smoothly for anyone. Systemic racism in action I guess.