338

Slrpnk.net outage (resolved) (feddit.org)

submitted 2 weeks ago* (last edited 6 days ago) by Kris@feddit.org to c/fediverse@lemmy.world

122 comments fedilink hide all child comments

Edit: unexpectatly we are back online. Will share how we found a solution later. There might be still some federation issues though.

Alt account of poVoq@slrpnk.net here.

Our instance is currently down and I can't get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can't work around remotely.

I am still trying a few things, but I am not very optimistic that I can get access.

The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can't access it physically during the next few weeks and my usual back up that could restart it is not available either.

As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy's law seems inescapable 😓

I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing 😥

Edit: we might use this "opportunity" to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.

(page 2) 50 comments

sorted by: hot top controversial new old

[-] khaliso@lemmy.world 8 points 1 week ago

Thanks for the heads up! My Main account is in slrpnk, so I'll set a reminder in an month to switch back. :)

[-] khaleer@sopuli.xyz 8 points 1 week ago* (last edited 1 week ago)

It looks like there is no enough solar power for solarpunk!

~~no need to show me where doors are, I will find them by myself~~

anyway, good luck with that!

[-] perestroika@lemm.ee 8 points 1 week ago* (last edited 1 week ago)

Thanks for the info, a healthy amount of real life awaits then. :)

I knew it was running on solar energy and old hardware so I guessed something like this had happened.

If you need fail-over to awaken a backup system when the primary fails, things can be designed. :)

[-] originalucifer@moist.catsweat.com 7 points 1 week ago

people hate the big cloud solutions, but this is the kind of thing their HA infrastructure prevents against... hardware failures

i dont enjoy using (or paying for) aws, but i will never have a firewall or disk failure.

[-] drspod@lemmy.ml 7 points 1 week ago

Their HA infrastructure is all built on open source projects. The thing they have that we don't is teams of SREs on-call 24/7.

[-] match@pawb.social 7 points 1 week ago

Ty for the update! slrpnk is my favorite instance that isn't my home instance

[-] PM_ME_VINTAGE_30S@lemmy.sdf.org 6 points 1 week ago

That sucks. Thank you for your time and running the instance. Gonna miss you all like hell while you're gone. 😞

[-] tenchiken@lemmy.dbzer0.com 6 points 1 week ago

Sending positive vibes, and passing offer of technical assistance once you get access again.

Hopefully it's an easy fix once you get into things again.

[-] LeighWeighTacoma@lemmy.world 5 points 1 week ago

I appreciate the update was thoroughly confused

[-] silence7@lemmy.world 5 points 1 week ago

Thanks for letting us know what is going on.

[-] Kris@feddit.org 5 points 1 week ago

Sorry that it took so long to get the word out. I guess we relied a bit too much on the xmpp server for internal communication.

[-] silence7@lemmy.world 6 points 1 week ago

Even the big operations break in unexpected ways. Facebook accidentally broke the authentication for their door locks when they took the site down a few years back and had to fly people to a datacenter and break down a door to get to servers and restart things.

load more comments (1 replies)

load more comments