74

On June 27:th, the https certificate for lemmy.today was supposed to renew itself. Its set up to automatically renew every 3 months, and its been working perfectly since the site was started over 2 years ago.

But this time it failed to do so, due to a syntax error in the web server config that was not noticed until this incident took place.

Its been fixed now but took more than a day unfortunantly. Sincere apologies for the long delay - it was because I was on vacation in Madeira and in the process of travelling home that day.

Life finds a way to kick us in the balls sometimes.

top 20 comments
sorted by: hot top controversial new old
[-] Allero@lemmy.today 13 points 3 months ago

Thanks! Happy to know it got resolved, and I hope you had a nice vacation!

[-] mrmanager@lemmy.today 11 points 3 months ago

It was very nice and very much needed to get a break from work for a while. Thanks for your understanding!

[-] DearMoogle@lemmy.today 9 points 3 months ago

Yayyy we’re back up! Thanks for the update. And we totally understand that you have a life outside of here lol :p

[-] sudo@lemmy.today 7 points 3 months ago

Enjoy the vacation, appreciate you so much (:

[-] YiddishMcSquidish@lemmy.today 7 points 3 months ago

Was really worried for a bit that y'all just shut the instance down. I was thinking "at least give me time to migrate my Furryosa pics!!!"

[-] mrmanager@lemmy.today 10 points 3 months ago

We will never shut it down just like that... It would be incredibly rude and insensitive towards you guys and we just dont operate that way as people.

This place is supposed to be very different from big tech platforms, and that means we see users as people to start with.

See you guys around!

[-] YiddishMcSquidish@lemmy.today 4 points 3 months ago

Chad energy having mfing mod!

[-] Walican132@lemmy.today 4 points 3 months ago

Thanks so much man.

[-] sorter_plainview@lemmy.today 4 points 3 months ago

When I got certificate error, I got really surprised. Nowadays no one does it manually. That was the only part made me a little bit concerned. Thanks for having this instance. I really appreciate the effort.

[-] Paradachshund@lemmy.today 4 points 3 months ago

Understandable. Have a nice day!

[-] tal@lemmy.today 3 points 3 months ago

Thanks for looking at it on your vacation! Sorry it came up then!

[-] lazynooblet@lazysoci.al 3 points 3 months ago

May I suggest signing up for hetrixtools.com

Free monitoring service that would have alerted you some weeks before the certificate expired.

[-] mrmanager@lemmy.today 1 points 3 months ago

Its an auto-renewing cert so it "always" works, but in this case there was a syntax error in the web server config, causing it to fail to actually reload the new cert. Will see what we can do to make that not happen again!

[-] lazynooblet@lazysoci.al 2 points 3 months ago

That cert is valid for 90 days at a time and is renewed every 60 days. So having an alert if the current certificate has less than 28 days remaining will give you an early warning that something went wrong with automation.

[-] mrmanager@lemmy.today 2 points 3 months ago

Ah yeah, thats a good idea. Thanks!

[-] hddsx@lemmy.ca 2 points 3 months ago

Let’s encrypt?

[-] OfficeMonkey@lemmy.today 2 points 3 months ago

I was worried, so I checked the error. And it made me confident something had gone wrong with autorenewal.

Given that my job had a days-long partial outage due to CLOCK SKEW between nodes, a cert renewal issue is positively understandable.

Glad you were able to get it sorted out, but hope you have some more vacation to get back to!

[-] AlecSadler 5 points 3 months ago

Oh man, clock skew between servers totally screwed me about two years ago because the servers were load balanced traffic and applied their server timestamp to realtime data - which in some cases led to entries being out of order. (never mind all the other issues with this implementation)

[-] ubergeek@lemmy.today 1 points 3 months ago

Thanks for fixing it, sorry it came while on vacation

[-] sic_semper_tyrannis@lemmy.today 1 points 3 months ago

I too thought maybe the instance shut down. I'm glad it didn't. No worries, I hope your vacation went well. I'll survive a few days without social media

this post was submitted on 28 Jun 2025
74 points (100.0% liked)

Announcements

314 readers
1 users here now

Announcements about system updates or other things related to this instance.

founded 2 years ago
MODERATORS