627
Well, shit. (lemmy.world)

256,338 rows affected.

you are viewing a single comment's thread
view the rest of the comments
[-] WhatsHerBucket@lemmy.world 59 points 2 days ago

Every seasoned IT person, devOps or otherwise has accidentally made a catastrophic mistake. I ask that in interviews :D

[-] partial_accumen@lemmy.world 42 points 2 days ago

Mine was replacing a failed hard drive in array.

  • Check array health, see one failed member
  • popped out the hot swappable old drive , popped in the new one
  • Check array health to make sure the array rebuild is underway
  • See array now has TWO failed member, and realize I feel the drive in my hand still spinning down

shit.

[-] WhatsHerBucket@lemmy.world 12 points 2 days ago

I accidentally rm’ed /bin on a remote host located in another country, and had to wait for someone to get in and fix it.

[-] manny_stillwagon@mander.xyz 11 points 2 days ago

Not IT but data analyst. Missed a 2% salary increase for our union members when projecting next year's budget. $12 million mistake that was only caught once it was too late to fix.

[-] LordOfLocksley@lemmy.world 15 points 2 days ago

I pushed a $1 bln test trade through production instead of my test environment.... that was a sweaty 30 minutes

[-] pticrix@lemmy.ca 9 points 2 days ago

I once deleted the whole production kubernetes environment trying to fix an update to prod give awry, at11pm. My saving grace was that our systems are barely used between 10pm-8am, and I managed to teach myself by reading enough docs and stack overflow comments to rebuild it and fix the initial mistake before 5am. Never learned how to correctly use a piece of stack that quickly before or since.

[-] martinb@lemmy.sdf.org 7 points 1 day ago

Nothing focuses the mind more than the panicked realisation that you have just hosed the production systems

[-] piefood@feddit.online 5 points 2 days ago

I deleted all of our DNS records. As it turns out, you can't make money when you can't resolve dns records :P

[-] Botzo@lemmy.world 8 points 2 days ago

Yep. Ran a config as code migration on prod instead of dev. We introduced new safeguards for running against prod after that. And changed the expectations for primary on call to do dev work with down time. Shifted to improving ops tooling or making pretty charts from all the metrics. Actually ended up reducing toil substantially over the next couple quarters.

10/10 will absolutely still do something dumb again.

this post was submitted on 06 Sep 2025
627 points (100.0% liked)

Lemmy Shitpost

34376 readers
4287 users here now

Welcome to Lemmy Shitpost. Here you can shitpost to your hearts content.

Anything and everything goes. Memes, Jokes, Vents and Banter. Though we still have to comply with lemmy.world instance rules. So behave!


Rules:

1. Be Respectful


Refrain from using harmful language pertaining to a protected characteristic: e.g. race, gender, sexuality, disability or religion.

Refrain from being argumentative when responding or commenting to posts/replies. Personal attacks are not welcome here.

...


2. No Illegal Content


Content that violates the law. Any post/comment found to be in breach of common law will be removed and given to the authorities if required.

That means:

-No promoting violence/threats against any individuals

-No CSA content or Revenge Porn

-No sharing private/personal information (Doxxing)

...


3. No Spam


Posting the same post, no matter the intent is against the rules.

-If you have posted content, please refrain from re-posting said content within this community.

-Do not spam posts with intent to harass, annoy, bully, advertise, scam or harm this community.

-No posting Scams/Advertisements/Phishing Links/IP Grabbers

-No Bots, Bots will be banned from the community.

...


4. No Porn/ExplicitContent


-Do not post explicit content. Lemmy.World is not the instance for NSFW content.

-Do not post Gore or Shock Content.

...


5. No Enciting Harassment,Brigading, Doxxing or Witch Hunts


-Do not Brigade other Communities

-No calls to action against other communities/users within Lemmy or outside of Lemmy.

-No Witch Hunts against users/communities.

-No content that harasses members within or outside of the community.

...


6. NSFW should be behind NSFW tags.


-Content that is NSFW should be behind NSFW tags.

-Content that might be distressing should be kept behind NSFW tags.

...

If you see content that is a breach of the rules, please flag and report the comment and a moderator will take action where they can.


Also check out:

Partnered Communities:

1.Memes

2.Lemmy Review

3.Mildly Infuriating

4.Lemmy Be Wholesome

5.No Stupid Questions

6.You Should Know

7.Comedy Heaven

8.Credible Defense

9.Ten Forward

10.LinuxMemes (Linux themed memes)


Reach out to

All communities included on the sidebar are to be made in compliance with the instance rules. Striker

founded 2 years ago
MODERATORS