This isn't exactly true unless you consider a pelvis that is incompatible with birth to be a neutral feature wrt to health. It is unlikely to kill you in an industrialized nation in the past 100 years, but outside of that environment or timeframe it is quite deadly for a woman to get pregnant while having a pelvis with an outlet like a male's. The baby always dies in such a situation and if the mother isn't killed she is disabled for life. While anthropologists use a lot of different measures to determine the sex of skeletal remains, the pelvis remains an extremely reliable indicator for exactly that reason. Even with other causes of maternal death causing a disproportionate number of humans to die vs other primates while giving birth, the female pelvis works well enough to facilitate birth for species continuation without preventing walking upright, which is also advantageous to the species.
fwiw gorillas are one of the most sexually dimorphic of ape species, we are closer to chimps or bonobos, which are still dimorphic. The parts of the brain that determine a stranger's sex are much older than apes and are generally reliable measures of a stranger's sex, especially if you are talking about seeing a person face to face rather than judging off of pictures. I don't think trooning would have gotten popular without the internet partially because it is so obvious in real life. Social scientists have studied different animals and young children in their ability to determine the sex of people and it does not seem to be a difficult task. I wonder if autistic people have a deficit in that specific task? It is social in nature....
to be fair though, is installing an artificial dick socket the same thing as repairing or transplanting a functional organ? Some radical feminist once said something like "what SRS creates is as much a new asshole as it is a 'new vagina', since it doesn't perform any of the functions of the organ it is named after and only exists so a man can fuck it".