[-] jemikwa 22 points 4 months ago* (last edited 4 months ago)

I want to clarify something that you hinted at in your post but I've seen in other posts too. This isn't a cloud failure or remotely related to it, but a facet of a company's security software suite causing crippling issues.

I apologize ahead of time, when I started typing this I didn't think it would be this long. This is pretty important to me and I feel like this can help clarify a lot of misinformation about how IT and software works in an enterprise.

Crowdstrike is an EDR, or Endpoint Detection and Response software. Basically a fancy antivirus that isn't file signature based but action monitoring based. Like all AVs, it receives regular definition updates around once an hour to anticipate possible threat actors using zero-day exploits. This is the part that failed, the hourly update channel pushed a bad update. Some computers escaped unscathed because they checked in either right before the bad update was pushed or right after it was pulled.
Another facet of AVs is how they work depends on monitoring every part of a computer. This requires specific drivers to integrate into the core OS, which were updated to accompany the definition update. Anything that integrates that closely can cause issues if it isn't made right.

Before this incident, Crowdstrike was regarded as the best in its class of EDR software. This isn't something companies would swap to willy nilly just because they feel like it. The scale of implementing a new security software for all systems in an org is a huge undertaking, one that I've been a part of several times. It sucks to not only rip out the old software but also integrate the new software and make sure it doesn't mess up other parts of the server. Basically companies wouldn't use CS unless they are too lazy to change away, or they think it's really that good.
EDR software plays a huge role in securing a company's systems. Companies need this tech for security but also because they risk failing critical audits or can't qualify for cybersecurity insurance. Any similar software could have issues - Cylance, Palo Alto Cortex XDR, Trend Micro are all very strong players in the field too and are just as prone to having issues.
And it's not just the EDR software that could cause issues, but lots of other tech. Anything that does regular definition or software updating can't or shouldn't be monitored because of the frequency or urgency of each update would be impractical to filter by an enterprise. Firewalls come to mind, but there could be a lot of systems at risk of failing due to a bad update. Of course, it should fall on the enterprise to provide the manpower to do this, but this is highly unlikely when most IT teams are already skeleton crews and subject to heavy budget cuts.

So with all that, you might ask "how is this mitigated?" It's a very good question. The most obvious solution "don't use one software on all systems" is more complicated and expensive than you think. Imagine bug testing your software for two separate web servers - one uses Crowdstrike, Tenable, Apache, Python, and Node.js, and the other uses TrendMicro, Qualys, nginx, PHP, and Rust. The amount of time wasted on replicating behavior would be astronomical, not to mention unlikely to have feature parity. At what point do you define the line of "having redundant tech stacks" to be too burdensome? That's the risk a lot of companies take on when choosing a vendor.
On a more relatable scale, imagine you work at a company and desktop email clients are the most important part of your job. One half of the team uses Microsoft Office and the other half uses Mozilla Thunderbird. Neither software has feature parity with the other, and one will naturally be superior over the other. But because the org is afraid of everyone getting locked out of emails, you happen to be using "the bad" software. Not a very good experience for your team, even if it is overall more reliable.

A better solution is improved BCDR (business continuity disaster recovery) processes, most notably backup and restore testing. For my personal role in this incident, I only have a handful of servers affected by this crisis for which I am very grateful. I was able to recover 6 out of 7 affected servers, but the last is proving to be a little trickier. The best solution would be to restore this server to a former state and continue on, but in my haste to set up the env, I neglected to configure snapshotting and other backup processes. It won't be the end of the world to recreate this server, but this could be even worse if this server had any critical software on it. I do plan on using this event to review all systems I have a hand in to assess redundancy in each facet - cloud, region, network, instance, and software level.
Laptops are trickier to fix because of how distributed they are by nature. However, they can still be improved by having regular backups taken of a user's files and testing that Bitlocker is properly configured and curated.

All that said, I'm far from an expert on this, just an IT admin trying to do what I can with company resources. Here's hoping Crowdstrike and other companies greatly improve their QA testing, and IT departments finally get the tooling approved to improve their backup and recovery strategies.

[-] jemikwa 22 points 5 months ago

I looked it up, the scale is relative popularity.

Numbers represent search interest relative to the highest point on the chart for the given region and time. A value of 100 is the peak popularity for the term. A value of 50 means that the term is half as popular. A score of 0 means there was not enough data for this term.

[-] jemikwa 25 points 6 months ago

The subtitle of the article says it's not available in the US -

PC Manager app is only available in some regions, but could come to the US eventually

[-] jemikwa 24 points 8 months ago* (last edited 8 months ago)

Actually a great idea, considering how expensive rims and wheels can be

[-] jemikwa 18 points 9 months ago* (last edited 9 months ago)

This has been happening with my original MS email account for years. It's been in so many data breaches and pwns over the years that I basically have abandoned it. It's constantly being probed by malicious actors from outside the US. I still keep it for when family reaches out, otherwise I'd close the account.
There's no real way to block the attempts. Make sure your password is rock solid (randomize and store it in a password manager) and unique, put on 2FA, and ensure your recovery methods aren't easily phishable/leakable.

[-] jemikwa 19 points 9 months ago* (last edited 9 months ago)

If you're paying property taxes, you're going to be in a database. At least here in Texas, all addresses (home, business, empty land) are in the county's appraisal and tax database that's publicly searchable.

[-] jemikwa 18 points 11 months ago

This looks to be Google ending support for the Android Auto framework on older Android versions, that's all. It's not about the car, it's about the phone.

[-] jemikwa 20 points 11 months ago* (last edited 11 months ago)

Technology Connections makes very specific, detailed videos on tech stuff, appliances, etc. My recent favorite of his is about incandescent vs LED Christmas lights and his unending rage about LED lights looking like "a computer threw up on your lawn". The first I remember watching was about dishwasher tech and how great they are, but how terrible dish pods are.

[-] jemikwa 24 points 11 months ago

It's a discord reaction of the nerd emoji. Supposed to imply the snake is calling her a nerd for following what they heard

[-] jemikwa 21 points 1 year ago

My Nvidia card says no to Wayland+KDE :( incredibly laggy and unresponsive ui

[-] jemikwa 19 points 1 year ago

I'd hate to be the admin that has to administrate that monstrous o365 tenant

[-] jemikwa 22 points 1 year ago* (last edited 1 year ago)

From an IT perspective with little context on this change other than what's in the article, if there's no way to import your own certs using an MDM, this change is terrible for businesses.

You need custom certs for all kinds of things. A company's test servers often don't use public CA certs because it's expensive (or the devs are too lazy to set up Let's Encrypt). So you import a central private CA cert to IT-managed devices so browsers and endpoints don't have a fit.

For increased network security, private CAs are used for SSL decryption to determine what sites devices are going to and to check for malware embedded in pages. In order to conduct SSL decryption, you need your own private CA cert for decrypting and re-encrypting web content. While this is on the decline because of pinned certs being adopted by big websites, it's still in use for any sites you can get away with. You basically kill any network-level security tools that are almost certainly enabled on the VPN/SASE used to access private test sites.

view more: ‹ prev next ›

jemikwa

joined 1 year ago