That's not quite right. In sci fi there are rules that remain logically consistent, based on known physics or not. In fantasy, they can make up the rules as they go along.

Compare Star Trek and Star Wars. Trek's subspace field theory is not any kind of known physics, but Trek is still sci fi. Because the subspace fields are fairly consistent about what they do. Star Wars' fan base damn near revolted when the series hinted that The Force might not be purely mystical.

Sci fi doesn't have to explain what the rules are to the reader. They just have to be be discoverable using the scientific method. Dragonriders of Pern is hard science fiction, originally published in Analog.