While I think Sam brings in a lot of viewers, he is not the reason people keep watching. If they said that in season 5, Sam wasn't going to be as involved, I don't think fans would stop watching. Sam was barely involved this season, completely zoned out when on screen, and people were still loving it.
However the show would still need a host with presence, something that Jet doesn't have, and Ben lacks the stature for. I read someone saying that Luke could do it, and I think that with a little bit of practice he probably could.
Sam could still be the main host, but only show up on the first couple of days and then once a week or so, until the last challenge, while the other host takes up his job with the daily activities.