Take a look at the concept of “spiral campaign development” in which you focus on the characters and where they are right now and build out from there.

https://slyflourish.com/spiral_campaign_building.html

First rule of Worldbuilding: Don’t make everything all at once.

It’s hard to resist sometimes, but if you focus too much on the large-scale, you will soon feel stretched too far. Instead, try working in smaller increments. Go to a particular area in your world. What is it like? What has happened in the last 100 years? The last 1,000 years? If you do it this way, not only is it more manageable, but it also allows you to make each individual area more interesting and unique.

Second rule of Worldbuilding: Remember to make a timeline!

Having a set timeline of your world, or even just for a city or country, can make life so much easier. It can also answer questions. For example: “Why is ______ city so poor, with almost no military protection?” Answer: “Because a decade ago, a plague swept over it, killing many of the towns people, including merchants and guards.”

Third and final(for my list) rule of Worldbuilding: Nothing is set in stone- until it is.

Let mw explain. Certain parts of your world, how it works, and how it happens are set in stone- they are vital to running the world as a whole. But as a DM with players, you must let their actions also control the world. A world seems easier to run if all aspects are made already, sure. But it makes the players feel as if their actions have no impact. For example if the party defeats a corrupt king, who would take his place afterwards? If the party removes a curse, letting the drought over a land vanish, what would happen? Let the players mould the present in your world- it should not be immutable. (If time travel is involved, then try adding consequences to it as well.)

I hope my three rules help you out! Good luck, worldbuilder!

I can tell you something that is working remarkably well for me, and entirely by accident: I had a sketch of a campaign world I wanted to build, but only 1 small area that was solid. It languished for a couple of years.

Then I got introduced to a website where people publish serial novels (Royal Road).

A few months later I had a bunch of story ideas bouncing around in my head, and a particular combination was demanding most of my attention, so I started writing it. I'm a bit of a 'panster', so I just started with my opening scenario and wrote.

But I found myself needing information for the world around them really fast, and then realized I had a solution: It fit perfectly in the campaign world I had started. This means that to keep writing the story, I had to keep filling out details that could be added to my campaign website. I also have two other story ideas I have started to sketch out that help fill details on other continents.

In short: If you can write a story in your campaign world, then answering questions around that story will give you campaign details.

Important note: The story you write IS NOT and WILL NOT be the adventure your players go through. Ditch that idea immediately. The story is its own thing.

For worldbuilding I recommend starting with broad strokes, then fill things in with tiny details first and grow out from there as needed. That may sound weird. What I mean is start with the big picture. Is your world a spherical planet or a flat Earth type situation, possibly something like Discworld where it's being carted around space (or the Astral Sea) by some giant elephants riding an even bigger turtle? Once you have that, scale down a bit one step. Are there multiple biomes/climates like deserts, jungles, temperate forests and savannah plains, or all a giant water world with scattered islands as the only landmass populated by mermaids, pirates, and one guy that uses a Simic Hybrid stat block and looks suspiciously like Kevin Costner?

Now think of the type of the story you want to tell and what the world needs for that. This may actually supersede the previous steps if it requires specific fundamental elements of the entire world but we're still working a mostly blank canvas inside of that frame you have so far. Tying in with "what about when players do something unexpected?" I recommend not planning the story too much ahead of time. I like to think of it like how professional wrestlers (who are really just theater nerds who also like getting jacked and doing all their own stunts) prepare for a big showcase match. They usually plan a few "spots" which are the key turning points of the narrative in their match. These spots are also usually what shows up most on highlight reels, big flashy moves like somebody getting thrown out of the ring, a particular high flying move, a third party interfering, and of course the big finish. They mostly improv all the action between the spots (sometimes they only have the end planned and improv the entire match but unless you're looking to run a pure sandbox game and just pull everything out of your ass as you go you should at least have something planned for your next session). Plan your spots as key events to advance the story. Make sure to drop plenty of hints and story hooks so your players know where to go for the next spot to happen. Plan for what you want them to do, anything you think they might do differently, and still try to leave wiggle room to improvise because they just might do something completely different. Start filling in the details around those spots, starting with the very beginning of the game. What's your hook to interest the PCs in the main story? Where is it? A tavern, a castle, a guild hall, or some empty field in the middle of nowhere that they all happen to be in for some reason? Then build out from there until you have a framework for that spot with necessary NPCs and locations like shops, taverns, dark forests, spooky caves, and haunted castles. Then other details like towns, cities, nations, and such that these places can be found in. But remember that you're going for a finely detailed picture frame to let your players paint within by their decisions because this is a game of collective storytelling and you're like the conductor of a symphony who is leading the team but not actually playing an instrument. Okay mixed metaphors but hopefully you get it. Then once you've figured out what they're doing you make adjustments to your next spot as necessary and drop hints, hooks, leads, etc to lead the party to that. Fill in the canvas between your frame as you go along with the help of your musicians. Sometimes you might have to change or move the next spot to fit with the way they reacted, like deciding an oval shaped frame is better than the square you started with for what's now a painting of a tuba instead of a piano. Also you're now out of brass colored paint, maybe use some of the leftover wood tones, ivory, and ebony for the next frame (this means if they skipped an NPC or plot twist you intended to be important just move it elsewhere and work it into the next segment of the story).

As to players going "wildly off course" that could be a few different things. First, are you all looking to be playing a game around the concept of linear storytelling (what I just described is intended to be a flexible approach to that) that follows a single cohesive and continuous narrative? Or is that player running off chasing butterflies or whatever wanting to play a sandbox game where everybody just does whatever strikes their fancy at the moment and expect you as DM to make stuff up on demand? Sometimes a player will say they want the latter but really just take pleasure in actively avoiding anything the DM provides and those people are sadistic assholes. I don't like to play with those. If this kind of thing happens often and you don't think they're just being dicks but rather just missing or misinterpreting the hints and leads you've presented, I recommend just talking to them about it. Admit that they are way off what you planned for and so far outside the lines you can't even hear what instrument they're playing. Or something like that I'm tired and fixated on mixed metaphors now. Very specifically ask them what they thought about you telling them the things that you intended to be hints and clues pointing at the stuff you have prepared and figure out why they missed them. It could be you not making things as obvious as you thought, or maybe your red herring was too smelly, or they're just dense and completely missed obvious clues. Unless they're just being a troublemaking jerk who brought a vuvuzela to the orchestra it's probably actually some combination. Talk with them and figure out how to more effectively communicate so everybody has fun and stays at least close enough to what you have planned and the music of your story doesn't stop with the sound of a record scratch because someone is expecting a DJ to drop beats for the freestyle rap of a sandbox game.

Okay I think I might have overdone the metaphors there but I'm tired and found it amusing. Hopefully some of that made enough sense to be helpful. Maybe take it as an example on improvising weird shit that just popped into somebody's head so that it fits with the story you had planned. Because I'm now insisting that the mixed metaphors tacked onto the advice I intended to give from the start are themselves metaphors for player generated chaos. That's my story and I'm sticking to it.

My two biggest suggestions are to focus on the local area and build outward as needed, and to know what your central tension is.

The first one is simple enough. Building a town and a goblin cave is easier than a planet and the realm of super magical evilness.

For the second one, central tension can be thought of as an underlying theme in your world. For example the main tension in my world is safety and stagnation vs uncertainty and progress. The gods keep us safe,, but that also means we have no need to struggle and grow. Even though the players aren't really aware of that even being a theme of the game yet, just knowing it myself can inform what kinds of characters and plots I set up as I continue to build more. When you know what the main conflict is at a more abstract level, you can see how certain characters might be on one side or the other without realizing.

Consider tying your narrative to atleast one of your characters back story's, it helps keep the players motivated. Past that just go with the flow and see where the current takes you.

Wildly off course can make some of the most memorable moments. Talk to your players often, see where they are interested, and where the characters are interested.

What I find helpful is knowing the broad strokes of what I want to present/create. What the overarching themes are, if there are some tropes I want to lean on, how I want it to feel. Having a feel for this on various levels (such as personal, local, cosmic etc) puts me in a focused mindset. I have the broad picture.

Details then. I try to keep most within the larger picture but also make sure there are a few things outside it to spice it up. And here I find creativity to be one of the best tools, just making things up. Random tables can be really useful, even if you don't roll on them. For example I make use of Ironsworn's (pdf is free) Character Goal table pretty much all the time. Sometimes the goal is defining of them, other times it just sits there being an annoyance to them. Can also recommend the tables from Worlds Without Numbers (basic pdf free).

Then you will need something to keep your work organized. I use kanka.io as I found it having less bloat than some other services such as worldanvil. Browse around and try things out. Papers in folders also work for many.

I seldom find myself In situations where I have no idea what is in the direction the party suddenly goes into. The broad picture is there for me to rely upon. But I know what you are getting at. When inspiration fails I find there are two great methods. First being random tables, roll a bit and see what the fates have for you. Second is to draw on your players for inspiration, ask them questions and use the answers (a gm "rule" from Apocalypse World, read it when you have the time). Ask them what they expect to find over there, what draws their characters there. Do they expect treasure? Adventure? Try and get their intention for going there. Often what you get is the broad outline for a new adventure and you just have to realize it with them.

Good luck and do ask follow ups if you have them.

Highly recommend looking up slyflourish and the series "lazy dungeon master". Helped me streamline prep a lot. Biggest things was coming up with the few things you want feature, understanding a general sequence of events and then tying them together along the way.

No session will survive contact with the party, and neither will a campaign. Being able to keep a loose framework you can work within throughout the campaign is the way everyone at the table can have fun.