I'm imagining this might come from way back when it was common for buildings to just be walls and a roof, and the ground floor was literally just the ground. Then the second level, if there was one, would be the first time they actually built a floor.