My wife's sister is either considering identifying as a man, or is identifying as gender fluid, we're not sure which as it's very touchy [...] She is also extremely rebellious
Without demanding powerleveling, there's some clarifying information that might help.
Is your sister-in-law still living with your wife's parents, or geographically much closer to them than you? It sounds like this information is second-hand through someone who didn't really get what was going on either. It seems unlikely that "Call me Milo now" didn't come with a long and detailed explanation of what everyone else's duty was to her mental health.
It also seems odd that parents who are cold and distant (and just allowed sister-in-law's "tumultuous life" to happen) would be the type to suddenly go all-in on support when it's troonery. The reason they cut off some of their own family would possibly help to explain; was this a reasonable action, and what flavor was the offense.
You're right: this is going to be no fun at all to navigate.
I think a good starting point is to clarify your position, with yourself as well as with your wife. (Hopefully you're on the same page, but agreeing to protect your kid makes it sound like you are.) Your sister-in-law is slapping the trans Band-Aid over an older, bigger problem, which is bad for her. Your sister-in-law is possibly about to start demanding everyone in the family change
for her, and if the parents are playing along out of convictions or social pressure, this is an unprecedented amount of power for a kid who grew up feeling neglected to suddenly feel, and it's going to reinforce that "gender euphoria" that's really just the sparkle of a new special interest and a new automatic friend group.
Being already aware of the common arguments of troonery can only help. If you're able to articulate your thoughts without sounding angry, if you're able to frame it as concern for her and the family, that will help. A lot of this comes as a surprise to people who aren't from the Internet, so they'll accept what a troon says as they helpfully educate the normies on something they've never even heard of before, and haven't even had time to think about.
And if you need to go nuclear you can start sending her pooners.