Apologies after the update this morning the docker host running the lemmy frontends (seperate from lemmy itself) was having real issues and took a while to resolve.

All back up and running now :)

Weird issue though - there was an unused container running squid (the forward proxy service) that was causing the system to freeze. The proxmox host node was getting a load average of 600+. Took a while to find it as had to start each container one, wait 2 minutes for it to settle, then try the next.

There shall be a 30 minute downtime after this post is 30 minutes old. ~ 12:40 UK time

The Proxmox cluster Lemmy is running on is being updated to version 9, and I'm doing all nodes at pretty much the same time, so all VMs are going off for the duration.

Update: All back up and working

Imgur have made the decision to block the UK following being threatened by the UK IPO due to the Online Safety Act.

Unfortunately this Lemmy server resides in the UK and is not unable to access imgur as a result. The image failures you may be expieriencing is due to this instance fetching images on your behalf to protect your internet identity (image_mode: ProxyAllImages)

I am working on a work-around which is almost ready to deploy. I expect images to stop functioning completely for a short while whilst I migrate pict-rs to another system (thank god I used S3 for pict-rs storage, else this migration would take days).

Update: pict-rs wasn't migrated, instead its outbound traffic has been routed out of another region. Unfortunately attempting to purge the cache of imgur images so it'll redownload is too much of a headache. The cache is set to 7-days but it seems setting it to 1-hour doesn't affect existing images, only new ones.

If you are in the UK, then new posts that use imgur.com images shall be viewable. However this will depend on your client. For example Boost for Android gets the image URL direct from the original server, so I see the the thumbnail OK (cached via lazysoci.al) and then the "imgur has blocked you" full image (due to Boost going direct).

I expect this will only get worse.

Typing this from my phone as I'm out cruising on the bike :) it's gonna rain soon so planning to do some updates when I get home later today.

I'll upgrade to latest Lemmy, I think we are 2 versions behind.

I'll also relax the image dimensions and movie size restrictions for the image proxy which I've noticed have prevented some image posts from working.

I don't know exactly what time it'll be but sometime shortly after 1300 UK

Update 2025-09-14 10:01: Real-life got in the way of the update yesterday. Updates have now been applied and we are now on 0.9.13. Also updated frontends and added blorp at https://b.lazysoci.al/ (thanks @nord@lazysoci.al)

Update 2025-09-14 10:29: Removed additional restrictions on dimensions of video uploads for the image proxy, this has resolved at least one example of the "too wide" error received for some images. It is now using the defaults, will see if there are any other examples and adjust if needed.

When this post is ~90 minutes old, lazysoci.al will have a downtime of ~1 hour.

I am upgrading the cluster that hosts lemmy, upgrading both RAM and Proxmox on all nodes.

Update: 1230 GMT+1; All back up and running.

but @LadyButterfly is killing it in my feed.

lazysoci.al comes under immense pressure at times due to bots scraping the site. As all data is federated, they can essentially target any instance, small or big and scrape most of lemmy.

Usually the server hosting lazysoci.al can keep up with it. Sometimes, like today, the response times of the site suffer and it triggers an alert.

If I'm at my PC (which is like 18 hours a day /nerd) then I'll add another rule to Cloudflare to target the bots. Today the response was to block Tencent's AS#, which is a LOT of IP addresses. I'll check later today if I've snagged a Lemmy instance on this range and make adjustments.

Today was slightly special though, as the sheer number of unique IP addresses was 781 (!!). Any sort of rate limiting on a per-IP/visitor basis is not going to get triggered when they have this number of IPs to play with.

Usually Lemmy doesn't use much RAM, but today the server had to use its swap-space due to the number of concurrent requests. So I am going to double the RAM of the server in a few minutes.. Downtime will be short, likely go unnoticed.

I shall shortly be updating Lemmy to 0.19.11 which fixes a security issue. Downtime is expected to be minimal.

At the same time I shall also be updating all alternative frontends to their latest versions.

Update

Upgrade completed, we are now on Lemmy 0.9.11! 👍 The update took 6 minutes, which is 5 minutes longer than I had expected. The update included a refresh of database indexes which wasn't documented in the upgrade notes. Sorry for the inconvenience if you tried to access during that period.

Along with the upgrade to 0.19.10 today I have made the following additional changes

• Added Photon frontend
  • Based on the project at https://github.com/Xyphyn/photon
  • Available at
    • https://photon.lazysoci.al/
    • https://p.lazysoci.al/
• Added Voyager frontend
  • Based on the project at https://github.com/aeharding/voyager
  • Available at
    • https://voyager.lazysoci.al/
    • https://v.lazysoci.al/
• Added Alexandrite frontend
  • Based on the project at https://github.com/sheodox/alexandrite
  • Available at
    • https://alexandrite.lazysoci.al/
    • https://a.lazysoci.al/
• Added Tesseract frontend
  • Based on the project at https://github.com/asimons04/tesseract
  • Available at
    • https://tesseract.lazysoci.al/
    • https://t.lazysoci.al/
• Removed the Lemmy UI gearbox debug icon
  • Thanks for pointing this out @Blaze@lemmy.dbzer0.com @Blaze
• Created this community
  • Another blaze idea, keep them coming :)