Because that would allow an ongoing process of just appointing a new Speaker pro tempore by the current Speaker (or current pro tempore), then voting to remove the current Speaker and let the new tempore take over, etc, so the Speaker is never voted in by a majority by both sides. It’s an intentional limit to make sure the current party cannot just keep passing the ball without input from the other side except to remove whoever is currently in the position.