The house can if they want. But presumably some plurality of the house would prefer a different speaker, so that vote will probably happen at some point in the future. It doesn't have to though.

Because that would allow an ongoing process of just appointing a new Speaker pro tempore by the current Speaker (or current pro tempore), then voting to remove the current Speaker and let the new tempore take over, etc, so the Speaker is never voted in by a majority by both sides. It’s an intentional limit to make sure the current party cannot just keep passing the ball without input from the other side except to remove whoever is currently in the position.