97
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 19 May 2026
97 points (100.0% liked)
Technology
84769 readers
3482 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
They’re probably stacks of 8x NPU Huawei servers all cooperatively serving the same few models.
As an older example, I believe Deepseek V3 was most optimally served with ~384 GPUs in a single cluster, before they switched to Chinese NPUs. So they’d have some software that ties all these together as one “server” and maybe multiple of those all serving API requests for one endpoint.
But it doesn’t actually need all 384 in each server. Many models will fit in a single 8-GPU/NPU server, but the software pools more just to try and utilize the hardware better.
If one server fails, the system would return a few requests as empty and have to restart the serving software, but… that’s fine. All the data is ephemeral. Even if the whole 24MW unit fails, they can just route API requests somewhere else, and a few failed generations isn’t a big deal.