How to run background jobs? (lemmy.world)

submitted 1 year ago* (last edited 1 year ago) by nothingness@lemmy.world to c/rust@lemmy.ml

13 comments fedilink hide all child comments

Is there any library for the queueing mechanism?

What's used by the most - Cron? But a task or rather script executed by Cron won't access to the context of an application. Meaning, a task will have be an independent unit. Whereas I want is a library to use inside a project such that it'll have access to everything.

Anything similar to Sidekiq exist in Rust?

you are viewing a single comment's thread
view the rest of the comments

[-] kevincox@lemmy.ml 1 points 1 year ago* (last edited 1 year ago)

For my project I just run them. I'm using async so it is just spawning a task in the background, but you can do the same with threads. Don't underestimate the number of threads that you can run on a modern computer.

If you want some sort of throttling you can stuff tasks into a queue and just run N background threads pulling them off an processing them.

If you need durability then you start to have more trouble. This is where I would start looking at a library. IDK if there are any libraries that handle logging, retries and similar, but if not you can probably get the basics down pretty easily.

[-] nothingness@lemmy.world 1 points 1 year ago* (last edited 1 year ago)

For my project I just run them.

How would you "just run" a task every 30 minutes? Every 5 hours? Once a day?

[-] kevincox@lemmy.ml 2 points 1 year ago

std::thread::spawn(|| {
    loop {
        std::thread::sleep(std::time::Duration::from_secs(30*60));
        do_job();
    }
});

Works pretty well. Maybe add a bit of code to crash the whole process on panic or some other logging. Wastes a few KiB of memory per loop but probably not a major issue. Doing this with async will waste only the tiniest amount of memory.

[-] nothingness@lemmy.world 1 points 1 year ago* (last edited 1 year ago)

what if one of the calls crashes? how would you re-run it?

[-] kevincox@lemmy.ml 1 points 1 year ago

Best option is probably to add a wrapper around the thread that re-spawns it. But you can also just catch panics in the loop.

[-] nothingness@lemmy.world 1 points 1 year ago* (last edited 1 year ago)

How would you re-run it multiple times then? An internal should be progressively greater. How would you terminate it if it continues to produce an exception?

[-] vest@mastodon.online 1 points 1 year ago

@kevincox How do you stop the job? Do you use channels like in Go?

[-] kevincox@lemmy.ml 1 points 1 year ago

It depends. Sometimes you can just put an exit call at the end of main to kill the thread. If you want to attempt graceful shutdown then usually I just use a boolean shutdown flag. Then the loop becomes while !shutdown.get() {

this post was submitted on 18 Sep 2023

11 points (100.0% liked)

Rust Programming

8150 readers

4 users here now

founded 5 years ago

MODERATORS

nutomic@lemmy.ml

Joe@lemmy.ml

AgreeableLandscape@lemmy.ml