482
AWS is having a bad day (health.aws.amazon.com)
top 50 comments
sorted by: hot top controversial new old
[-] db0@lemmy.dbzer0.com 253 points 1 month ago

It's wild that these cloud providers were seen as a one-way stop to ensure reliability, only to make them a universal single point of failure.

[-] Nighed@feddit.uk 135 points 1 month ago

But if everyone else is down too, you don't look so bad 🧠

[-] queerlilhayseed@piefed.blahaj.zone 67 points 1 month ago

No one ever got fired for buying IBM.

[-] cdzero@lemmy.ml 17 points 1 month ago

I wouldn't be so sure about that. The state government of Queensland, Australia just lifted a 12 year ban on IBM getting government contracts after a colossal fuck up.

[-] queerlilhayseed@piefed.blahaj.zone 55 points 1 month ago* (last edited 1 month ago)

It's an old joke from back when IBM was the dominant player in IT infrastructure. The idea was that IBM was such a known quantity that even non-technical executives knew what it was and knew that other companies also used IBM equipment. If you decide to buy from a lesser known vendor and something breaks, you might be blamed for going off the beaten track and fired (regardless of where the fault actually lay), whereas if you bought IBM gear and it broke, it was simply considered the cost of doing business, so buying IBM became a CYA tactic for sysadmins even if it went against their better technical judgement. AWS is the modern IBM.

[-] sugar_in_your_tea@sh.itjust.works 21 points 1 month ago

AWS is the modern IBM.

That's basically why we use it at work. I hate it, but that's how things are.

load more comments (1 replies)
load more comments (1 replies)
[-] clif@lemmy.world 13 points 1 month ago* (last edited 1 month ago)

One of our client support people told an angry client to open a Jira with urgent priority and we'd get right on it.

... the client support person knew full well that Jira was down too : D

At least, I think they knew. Either way, not shit we could do about it for that particular region until AWS fixed things.

[-] GissaMittJobb@lemmy.ml 57 points 1 month ago

It's mostly a skill issue for services that go down when USE-1 has issues in AWS - if you actually know your shit, then you don't get these kinds of issues.

Case in point: Netflix runs on AWS and experienced no issues during this thing.

And yes, it's scary that so many high-profile companies are this bad at the thing they spend all day doing

[-] village604@adultswim.fan 21 points 1 month ago

Yeah, if you're a major business and don't have geographic redundancy for your service, you need to rework your BCDR plan.

[-] Bob_Robertson_IX@discuss.tchncs.de 11 points 1 month ago
[-] village604@adultswim.fan 10 points 1 month ago

So does an outage, but I get that the C-suite can only think one quarter at a time

load more comments (1 replies)
load more comments (7 replies)
[-] tburkhol@lemmy.world 32 points 1 month ago

It is still a logical argument, especially for smaller shops. I mean, you can (as self-hosters know) set up automatic backups, failover systems, and all that, but it takes significant time & resources. Redundant internet connectivity? Redundant power delivery? Spare capacity to handle a 10x demand spike? Those are big expenses for small, even mid-sized business. No one really cares if your dentist's office is offline for a day, even if they have to cancel appointments because they can't process payments or records.

Meanwhile, theoretically, reliability is such a core function of cloud providers that they should pay for experts' experts and platinum standard infrastructure. It makes any problem they do have newsworthy.

I mean,it seems silly for orgs as big and internet-centric as Fortnite, Zoom, or forturne-500 bank to outsource their internet, and maybe this will be a lesson for them.

load more comments (3 replies)
[-] ms_lane@lemmy.world 11 points 1 month ago

They zigged when we all zagged.

Decentralisation has always been the answer.

[-] mhzawadi@lemmy.horwood.cloud 8 points 1 month ago

yeah, so many things now use AWS in some way. So when AWS has a cold, the internet shivers

load more comments (4 replies)
[-] Sunny@slrpnk.net 83 points 1 month ago

I hate how Signal went down because of this... Wish it wasn't so centralised.

[-] pineapplelover@lemmy.dbzer0.com 40 points 1 month ago* (last edited 1 month ago)

My friend messaged me on Signal asking if Instructure (runs on AWS) was down. I got the message. That being said, it's scary that Signal's backbone depends on AWS

load more comments (5 replies)
[-] herseycokguzelolacak@lemmy.ml 9 points 1 month ago

Signal's love affair with big tech is deeply disturbing.

load more comments (8 replies)
[-] jali67@lemmy.zip 57 points 1 month ago

Why do we place so much reliance on one mega company? This level of importance. It should be seized by the government.

[-] PM_ME_VINTAGE_30S@lemmy.sdf.org 40 points 1 month ago

It should be seized by the ~~government~~ people and mercilessly decentralized.

load more comments (2 replies)
[-] mhzawadi@lemmy.horwood.cloud 15 points 1 month ago

God no, not the government!

They couldn't organise a paper bag party

[-] jali67@lemmy.zip 35 points 1 month ago* (last edited 1 month ago)

Large corporations and oligarchs are better? I’ll take the government. At least we can vote on them.

[-] bss03@infosec.pub 13 points 1 month ago

I think co-ops are the way to go, but I can understand that someone "just" wanting to purchase the good/service might not see the difference between a co-op and corporation like Amazon.

I don't think it's a size issue really, but co-ops generally stay smaller in part due to how they are internally organized compared to a "median" corporation.

I also think that the government actually does a pretty good job at managing things; it's just their failures are public. Private boondoggles might drive many people into bankruptcy, but they aren't publicized any more than absolutely necessary.

load more comments (2 replies)
load more comments (7 replies)
[-] mic_check_one_two@lemmy.dbzer0.com 15 points 1 month ago

That’s largely because one half of the elected officials are dedicated to defunding and deconstructing government organizations, so they can then point at those same organizations and go “look, the government doesn’t work! We should stop funding it!” The government is actually great at organizing a lot of things. But they’re all so engrained in society that you don’t even think about them as being organized by the government. Systems that just work, reliably, all the time.

The government’s job is stability and reliability, not being as efficient as possible. Where a corporation may only have one person doing a job, the government will have four or five. Those people aren’t bloat; They’re on the payroll because the government is expected to keep functioning during emergencies. People would lose their minds if the streets department (responsible for clearing downed trees out of public roads) shut down after a bad storm rolled through, just because a few government employees had a tree branch fall on their house. What if firefighters stopped working because a local wildfire burnt a few firefighters’ houses? What if the city water department shut down because three or four city employees’ water supply was affected? What if the health department shut down during a pandemic?

The people who work in government also live in the same areas they serve. Which means that they are affected by the same emergencies. The government needs enough redundancy to be able to continue functioning, even after those employees are affected by the same emergencies as the general public. If some emergency affects 75% of the public in a given area, then 75% of the local government employees are likely going to be affected. So if the government doesn’t have enough redundancy to be able to redistribute the work, people will see their government shutting down in the wake of the emergency. And to make matters even worse, during (and in the wake of) those emergencies, people look to the government for help. Which means that’s the most critical time for the government to continue functioning.

I say all of this because the same is true for the infrastructure that runs critical government systems. The government expands and implements things slowly by design, because everything critical has to go through multiple levels of design approval, and have multiple redundancies built in. If the government has updated a critical system, I can guarantee that new system has been in the works for the past two years at least. That process is designed to ensure everything works as intended. I wouldn’t want my city traffic lights managed by a private company, because they’d try to cut costs and avoid building in redundant systems.

load more comments (1 replies)
load more comments (4 replies)
[-] noxypaws@pawb.social 15 points 1 month ago* (last edited 1 month ago)

AWS aggressively pursues high priced and years-long spending commitments with large customers, and they incentivize it with huge discounts for doing so.

And when AWS does this they intentionally incentivize these large customers to migrate existing workloads away from other cloud service providers as well, going so far as to offer assistance in doing so.

load more comments (5 replies)
[-] Alaknar@sopuli.xyz 15 points 1 month ago

Why do we place so much reliance on one mega company? This level of importance.

Because it's cheaper and (in broad terms) more reliable than everybody having a data centre.

It should be seized by the government.

Oh yeah, what could possibly go wrong if the US government owned Amazon!

load more comments (3 replies)
[-] Bluewing@lemmy.world 8 points 1 month ago

Do you really want someone like the magahats having control over something like that?

[-] Trying2KnowMyself 54 points 1 month ago
[-] chellomere@lemmy.world 7 points 1 month ago

This gif is audible

[-] Tolookah@discuss.tchncs.de 37 points 1 month ago
[-] HelloRoot@lemy.lol 20 points 1 month ago

You guys don't selfhost a registry?

[-] magguzu@lemmy.ml 10 points 1 month ago

I know this is selfhosted so most people here are hobbyists, but it's a ton of work to selfhost in enterprise setting. I'd wager 90%+ of people using image registries are using Docker Hub, GHCR, or AWS ECR.

load more comments (2 replies)
load more comments (3 replies)
load more comments (4 replies)
[-] RecipeForHate1@lemmy.ml 32 points 1 month ago

A bad day for Jeff Bezos is a good day for all of us

[-] aarRJaay@lemmy.world 30 points 1 month ago

Who wants to bet Amazon gave AI full access to their prod config and it screwed it up.

[-] possiblylinux127@lemmy.zip 11 points 1 month ago

Or some engineer decide today would be a great day to play with BGP

load more comments (1 replies)
[-] Cyber@feddit.uk 25 points 1 month ago
[-] otter@lemmy.ca 15 points 1 month ago

Is there no way to check the doorbell video locally?

An Amazon employee misconfigures something and now your doorbell doesn't work

load more comments (8 replies)
[-] 30p87@feddit.org 24 points 1 month ago

And I'm having a very good day now :3

load more comments (2 replies)
[-] AllHailTheSheep@sh.itjust.works 22 points 1 month ago* (last edited 1 month ago)

according to that page the issue stemmed from an underlying system responsible for health checks in load balancing servers.

how the hell do you fuck up a health check config that bad? that's like messing up smartd.conf and taking your system offline somehow

[-] ayyy@sh.itjust.works 19 points 1 month ago

Well, you see, the mistake you are making is believing a single thing the stupid AWS status board says. It is always fucking lying, sometimes in new and creative ways.

load more comments (2 replies)
[-] domi@lemmy.secnd.me 20 points 1 month ago

That explains why my Matrix <-> Signal bridge was complaining about being disconnected.

load more comments (5 replies)
[-] herseycokguzelolacak@lemmy.ml 12 points 1 month ago

that is an understatement 😂

[-] regedit@lemmy.zip 9 points 1 month ago

This kind of shit will only increase as more of these companies believe they can vibe-code their way out of paying software devs what they are worth.

[-] Tuxxer@lemmy.world 9 points 1 month ago* (last edited 1 month ago)

For some reason I hear Gilfoyle pontificating about what he does

[-] aichan@piefed.blahaj.zone 8 points 1 month ago
load more comments
view more: next ›
this post was submitted on 20 Oct 2025
482 points (100.0% liked)

Selfhosted

53454 readers
376 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

  7. No low-effort posts. This is subjective and will largely be determined by the community member reports.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS