I suspect s1 and 2 they did rig a lot of the eliminations but they always tried to hide it a lot. For example, the PSAT test challenge could have easily been rigged against Sylvia because we never actually got to see the results, Sam just came up and said Sylvia lost. Sylvia was also most certainly the right choice to go home though regardless of what the real results were which is why I strongly suspect they were rigged.
Sometimes fans demanding a fish be kicked from the show is a good thing too, like in s2 with Brian. The wiggers made it very clear that Brian was their favorite of the s2 contestants post-Cole and didn't want to kick him, yet the fans ALL demanded he get kicked (seemingly no one in the s2 audience wanted Brian around anymore in a literal sense) and the wiggers fully obliged surprisingly. Brian was a good character to have around while Cole was in the tank because they were the main enemies of each other but without Cole, Brian made the show insufferable to watch. So it was actually a good thing they kicked Brian for no other reason than the fans hated his guts.
I think having more fluid eliminations like the first 2 seasons works better for the chaotic nature of the show and it can help safeguard against people like Binx, Payton, Sylvia, or Brian overstaying their welcome while making sure people like Mizzy, Simbal, or Letty aren't punished for acting outside of the group mentality and being competitive