Doctors regularly tell women at risk for breast cancer to minimize their soy consumption. Fuck with your brain and the juices that control how you feel, think, and act at your own peril, but small amounts are likely not significant. Meal replacement shakes? We all know how cosmo turned out.
It's a bit more nuanced than that, but yes, plants has been known to interfere with the endocrine system for a long time, and soy is one of the strongest one there is.
People focus on the word "phytoestrogen" because it has "estrogen" in it, but it's more about the isoflavones, which is a specific type of phytoestrogen that can actually bind to human estrogen receptors and is abundantly present in legumes such as soy.
For example onions also have phytoestrogen of course, but their is mostly the lignans type and not isoflavones; lignans do not appear to bind in humans to any significant degree. This is why when you see a study about "what does phytoestrogen does in humans" it's not useful because there are many types of phytoestrogens and you have to read the study to find out what plant they took the phytoestrogen from, and how much.
It's like a study on what protein ingestion does to human, but there is a world of difference between plant protein and animal protein.
There isn't much that is 100% conclusive on soy, but all signs indicate that it is most likely bad for you. I think you will not be surprised to learn that there is no one willing to finance and fight ethical committees in order to get a huge double-blinded study that could discredit one of the biggest American export, a food that is in pretty much everything processed and has a very strong vegetarian, globohomo, and farming lobby behind it.
I haven't eaten soy or any product with soy for that reason for years. Since I don't eat processed stuff with soy lecithin in it, I also coincidently stopped most goyslop since they put that stuff in most processed stuff. So it's a win-win.
In short, don't be a soyboy.