153
Broken emojo caused site outage, (self.main)
submitted 1 year ago by supakaity to c/main
26 comments fedilink hide all child comments

I have a process which copies emojos from the hajkey blahaj.zone site to our lemmy.

Last night (for me) it copied an invalid emojo into the database and it broke lemmy for hours while Ada and I were sleeping.

I've found the issue and fixed it now, so everything should be purring along once again, but for those who were left without the instance to use, I sincerely apologise.

blobcat, sorry

all 27 comments
sorted by: hot top controversial new old
[-] leigh 30 points 1 year ago

No worries, things happen — and you’re community-building volunteers, not an on-call rotation. 💜 I’m just glad we’re back now! Thanks for your efforts during these past 24 hours. 🏳️‍⚧️😊

[-] PaigePalisade 22 points 1 year ago

I'm going to imagine that :capybara_animated: was chewing on the server racks

animated, capybara

[-] ada 11 points 1 year ago

That checks out

[-] amber 19 points 1 year ago* (last edited 1 year ago)

animated, capybara thanks for the update! what emojo was it

[-] supakaity 28 points 1 year ago

It wasn't an actual emojo. The script processed the SQL header column names as an emojo and tried to add them. Unfortunately publicUrl is not a valid URL, so lemmy's /api/v3/site metadata endpoint started returning an error relative URL without a base instead of the JSON that the website was expecting and so the site just stopped working for everyone the next time it tried to load that url.

[-] bamboo 11 points 1 year ago

This coincided with me resetting my password, and I thought I brought down the site somehow

[-] wetnoodle 16 points 1 year ago

happy the best instance is back up!! crabrave

[-] smoldragon 8 points 1 year ago

Glad to be back and thanks for your hard work!

[-] Krrygon@sh.itjust.works 8 points 1 year ago

Excellent news! The site loads perfectly for me. I am still unable to log in for some reason, but that could just be a me thing.

[-] wetnoodle 8 points 1 year ago

clear your cookies/cache for this site, it should work after that

[-] Krrygon 9 points 1 year ago

It works! I am back in the blahaj zone. Feels good.

[-] LuckingFurker 5 points 1 year ago

Thank you for all you do, dear admins ❤️

[-] Wirlocke 5 points 1 year ago

I was worried that it got hacked again, since that just happened. Glad that Blahaj lives again!

[-] stercar 5 points 1 year ago

So glad it‘s up again. I was thinking it was something like that since it kept giving a parsing error when I was trying to log in. Good that it‘s fixed now.

[-] essi 4 points 1 year ago

It happens! Thanks for the writeup, and good work bringing things back online again :)

[-] Devious_Thoughts 4 points 1 year ago

Appreciate the fix, but I am having trouble uploading images, wonder if anyone else is having them?

[-] supakaity 2 points 1 year ago

Should be fixed now

[-] djstini 4 points 1 year ago

{"data":{"msg":"Couln't upload file, Error in DB, IO error: Os { code: 28, kind: StorageFull, message: "No space left on device" }","files":null},"state":"success"}

Hmm, still getting errors, for example when trying to upload profile images

[-] supakaity 2 points 1 year ago

Should be fixed now.

[-] alx 4 points 1 year ago

Welcome back to the Friends From Sweden and thank you to the admins

[-] Rozauhtuno 4 points 1 year ago

Mistakes happen, I'm just happy the instance is back up.

[-] moss 2 points 1 year ago* (last edited 1 year ago)

I can't log into the website now. It tells me I logged in successfully, then simply doesn't log me in. Jerboa still works fine tho

[-] jdp23 2 points 1 year ago

What browser are you using? I have that issue in Firefox, but not in Chrome.

[-] moss 1 points 1 year ago

Firefox yeah

[-] Xtallll 2 points 1 year ago

After the Hack they changed the secret key, you have to clear the cookies for Lemmy.Blahja.zone and re-login.

[-] moss 1 points 1 year ago

I already did that it's not working

this post was submitted on 10 Jul 2023
153 points (100.0% liked)

Blahaj Lemmy Meta

2266 readers
1 users here now

Blåhaj Lemmy is a Lemmy instance attached to blahaj.zone. This is a group for questions or discussions relevant to either instance.

founded 2 years ago
MODERATORS
blahaj
supakaity
ada