799

This isn't a gloat post. In fact, I was completely oblivious to this massive outage until I tried to check my bank balance and it wouldn't log in.

Apparently Visa Paywave, banks, some TV networks, EFTPOS, etc. have gone down. Flights have had to be cancelled as some airlines systems have also gone down. Gas stations and public transport systems inoperable. As well as numerous Windows systems and Microsoft services affected. (At least according to one of my local MSMs.)

Seems insane to me that one company's messed up update could cause so much global disruption and so many systems gone down :/ This is exactly why centralisation of services and large corporations gobbling up smaller companies and becoming behemoth services is so dangerous.

you are viewing a single comment's thread
view the rest of the comments
[-] Swarfega@lemm.ee 61 points 3 months ago

I've just spent the past 6 hours booting into safe mode and deleting crowd strike files on servers.

[-] allywilson@lemmy.ml 18 points 3 months ago

Feel you there. 4 hours here. All of them cloud instances whereby getting acces to the actual console isn't as easy as it should be, and trying to hit F8 to get the menu to get into safe mode can take a very long time.

[-] Swarfega@lemm.ee 6 points 3 months ago

Ha! Yes. Same issue. Clicking Reset in vSphere and then quickly switching tabs to hold down F8 has been a ball ache to say the least!

[-] Blank@lemmy.world 2 points 3 months ago

Just go into settings and add a boot delay, then set it back when you're done.

[-] Avatar_of_Self@lemmy.world 2 points 3 months ago* (last edited 3 months ago)

What I usually do is set next boot to BIOS so I have time to get into the console and do whatever.

Also instead of using a browser, I prefer to connect vmware Workstation to vCenter so all the consoles insta open in their own tabs in the workspace.

[-] possiblylinux127@lemmy.zip 3 points 3 months ago
[-] ArrogantAnalyst@infosec.pub 10 points 3 months ago

Since it has to happen in windows safe mode it seems to be very hard to automate the process. I haven’t seen a solution yet.

[-] Swarfega@lemm.ee 5 points 3 months ago

Sadly not. Windows doesn't boot. You can boot it into safe mode with networking, at which point maybe with anaible we could login to delete the file but since it's still manual work to get windows into safe mode there's not much point

[-] lengau@midwest.social 7 points 3 months ago

It is theoretically automatable, but on bare metal it requires having hardware that's not normally just sitting in every data centre, so it would still require someone to go and plug something into each machine.

On VMs it's more feasible, but on those VMs most people are probably just mounting the disk images and deleting the bad file to begin with.

[-] Swarfega@lemm.ee 4 points 3 months ago

I guess it depends on numbers too. We had 200 to work on. If you're talking hundreds more than looking at automation would be a better solution. In our scenario it was just easier to throw engineers at it. I honestly thought at first this was my weekend gone but we got through them easily in the end.

[-] Natanael@slrpnk.net 1 points 3 months ago

The real problem with VM setups is that the host system might have crashed too

this post was submitted on 19 Jul 2024
799 points (100.0% liked)

Linux

48047 readers
681 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS