I feel like Secret Sober had two benefits going for it which helped with viewer attention. First, they killed off contestants immediately so the eight players shrank down to an easier to follow number. Second, the show identified Lili Du as "the" secret sober and edited the show around that definition, so players had a contestant to focus on.

So, you add more effective contestants and constantly move the focus around and it becomes harder for viewers to follow. I also think the additional side quests weren't really done in a way to draw attention to those doing the side quests.