There is no "right" answer for this question - it all depends on how you define isekai.

Is it just a trope, or a genre? If it's a genre, which works should be considered the canon? Does being transported into another part of the same literary universe count as "being in another world"? Does the protag need to be from Earth, or a world resembling Earth? etc.

Personally I don't consider it isekai. For me isekai is a lot like a novel, manga and anime movement, and Dune is simply too far from said movement to be part of it:

• it's considerably older - from the 60s. Most things that I'd call isekai are from the 00s onwards.
• it's the product of another culture - Dune was written in USA; most isekais are Japanese, or at most Korean and Chinese
• the setting is completely different - the setting in most isekais is a swords-and-magic world, or perhaps a game-like world. Dune is a basically a sand hell.
• as @Moghul@lemmy.world said, the world where Paul ends is part of the same literary universe as the world that he's from. That's actually quite impactful as it means that people in Paul's world Caladan are fully aware of Arrakis, and it's part of the same "power structure" as Caladan.
• etc.

None of those things would be enough on itself to say "no! not isekai!", but when combined they make me think "perhaps we should classify Dune as something else".

But again, that's my personal definition. If you use a different definition, calling Dune an isekai might be sensible.

No. It's the same world, it's just that the world encompasses both Caladan and Arrakis, as well as well... the rest of the galaxy. It's the in-world equivalent of Paul traveling to another country where he's unexpectedly royalty.

By that logic Star Trek is an isekai.