1164
submitted 4 months ago* (last edited 4 months ago) by rxxrc@lemmy.ml to c/technology@lemmy.world

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It's all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We'll see if that changes over the weekend...

(page 2) 50 comments
sorted by: hot top controversial new old
[-] lustyargonian@lemm.ee 38 points 4 months ago

Linux and Mac just got free advertisment.

[-] LordWiggle@lemmy.world 25 points 4 months ago

The words 'Mac' and 'free' aren't allowed in the same sentence.

load more comments (5 replies)
[-] kamenoko@sh.itjust.works 34 points 4 months ago

AWS No!!!

Oh wait it's not them for once.

load more comments (1 replies)
[-] spacesatan@lemm.ee 34 points 4 months ago

CrowdStrike Holdings, Inc. is an American cybersecurity technology company based in Austin, Texas.

Never trust a texan

load more comments (1 replies)
[-] alphacyberranger@sh.itjust.works 32 points 4 months ago* (last edited 4 months ago)

One possible fix is to delete a particular file while booting in safe mode. But then they'll need to fix each system manually. My company encrypts the disks as well so it's going to be a even bigger pain (for them). I'm just happy my weekend started early.

load more comments (11 replies)
[-] Treczoks@lemmy.world 29 points 4 months ago

I was quite surprised when I heard the news. I had been working for hours on my PC without any issues. It pays off not to use Windows.

[-] wizardbeard@lemmy.dbzer0.com 37 points 4 months ago

It's not a flaw with Windows causing this.

The issue is with a widely used third party security software that installs as a kernel level driver. It had an auto update that causes bluescreening moments after booting into the OS.

This same software is available for Linux and Mac, and had similar issues with specific Linux distros a month ago. It just didn't get reported on because it didn't have as wide of an impact.

load more comments (8 replies)
load more comments (1 replies)
[-] jedibob5@lemmy.world 27 points 4 months ago

Huh. I guess this explains why the monitor outside of my flight gate tonight started BSoD looping. And may also explain why my flight was delayed by an additional hour and a half...

[-] Nachorella@lemmy.sdf.org 27 points 4 months ago

My company used to use something else but after getting hacked switched to crowdstrike and now this. Hilarious clownery going on. Fingers crossed I'll be working from home for a few days before anything is fixed.

[-] veam@lemmy.world 27 points 4 months ago

oh joy. can’t wait to have to fix this for all of our clients today…

[-] iturnedintoanewt@lemm.ee 18 points 4 months ago

You have no idea how much fun its being.

load more comments (3 replies)
load more comments (1 replies)
[-] Pudutr0n@feddit.cl 25 points 4 months ago

This is a better article. It's a CrowdStrike issue with an update (security software)

load more comments (2 replies)
[-] thearch@sh.itjust.works 20 points 4 months ago

Irrelevant but I keep reading "crowd strike" as "counter strike" and it's really messing with me

load more comments (2 replies)
[-] solomon42069@lemmy.world 20 points 4 months ago

Why is no one blaming Microsoft? It's their non resilient OS that crashed.

[-] blackn1ght@feddit.uk 33 points 4 months ago

Probably because it's a Crowdstrike issue, they've pushed a bad update.

[-] solomon42069@lemmy.world 24 points 4 months ago

OK, but people aren't running Crowdstrike OS. They're running Microsoft Windows.

I think that some responsibility should lie with Microsoft - to create an OS that

  1. Recovers gracefully from third party code that bugs out
  2. Doesn't allow third party software updates to break boot

I get that there can be unforeseeable bugs, I'm a programmer of over two decades myself. But there are also steps you can take to strengthen your code, and as a Windows user it feels more like their resources are focused on random new shit no one wants instead of on the core stability and reliability of the system.

It seems to be like third party updates have a lot of control/influence over the OS and that's all well and good, but the equivalent of a "Try and Catch" is what they needed here and yet nothing seems to be in place. The OS just boot loops.

load more comments (5 replies)
[-] Mikina@programming.dev 19 points 4 months ago

I see a lot of hate ITT on kernel-level EDRs, which I wouldn't say they deserve. Sure, for your own use, an AV is sufficient and you don't need an EDR, but they make a world of difference. I work in cybersecurity doing Red Teamings, so my job is mostly about bypassing such solutions and making malware/actions within the network that avoids being detected by it as much as possible, and ever since EDRs started getting popular, my job got several leagues harder.

The advantage of EDRs in comparison to AVs is that they can catch 0-days. AV will just look for signatures, a known pieces or snippets of malware code. EDR, on the other hand, looks for sequences of actions a process does, by scanning memory, logs and hooking syscalls. So, if for example you would make an entirely custom program that allocates memory as Read-Write-Execute, then load a crypto dll, unencrypt something into such memory, and then call a thread spawn syscall to spawn a thread on another process that runs it, and EDR would correlate such actions and get suspicious, while for regular AV, the code would probably look ok. Some EDRs even watch network packets and can catch suspicious communication, such as port scanning, large data extraction, or C2 communication.

Sure, in an ideal world, you would have users that never run malware, and network that is impenetrable. But you still get at avarage few % of people running random binaries that came from phishing attempts, or around 50% people that fall for vishing attacks in your company. Having an EDR increases your chances to avoid such attack almost exponentionally, and I would say that the advantage it gives to EDRs that they are kernel-level is well worth it.

I'm not defending CrowdStrike, they did mess up to the point where I bet that the amount of damages they caused worldwide is nowhere near the amount damages all cyberattacks they prevented would cause in total. But hating on kernel-level EDRs in general isn't warranted here.

Kernel-level anti-cheat, on the other hand, can go burn in hell, and I hope that something similar will eventually happen with one of them. Fuck kernel level anti-cheats.

load more comments (1 replies)
[-] umami_wasbi@lemmy.ml 19 points 4 months ago

No one bother to test before deploying to all machines? Nice move.

[-] huginn@feddit.it 21 points 4 months ago* (last edited 4 months ago)

This outage is probably costing a significant portion of Crowd strike's market cap. They're an 80 billion dollar company but this is a multibillion outage.

Someone's getting fired for this. Massive process failures like this means that it should be some high level managers or the CTO going out.

load more comments (6 replies)
load more comments (1 replies)
load more comments
view more: ‹ prev next ›
this post was submitted on 19 Jul 2024
1164 points (100.0% liked)

Technology

59612 readers
3117 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 1 year ago
MODERATORS