PSA: We're back, here's a post-mortem! (beehaw.org)

submitted 1 year ago* (last edited 1 year ago) by Lionir@beehaw.org to c/support@beehaw.org

71 comments fedilink hide all child comments

Hi Beeple!

Here's a vague version of events :

11PM EST: Lemmy.world got hacked
12:20AM EST: Blahaj.zone got hacked
12:25AM EST: I shut down the server
12:30AM EST: I make announcements to tell people about this
12:45AM EST: I have an idea of what the problem is but there is no fix
2:20AM EST: I go to sleep
8:50AM EST: The server is booted back up, steps are applied to mitigate issues (Rotating JWTs, Clearing DB of the source of vulnerability, deleting custom emoji), UI is updated with the fix, CSP and other security options are applied
11:40AM EST: We start testing things to make sure are working And well, now here we are.

If you have issues logging in or using an app:

Log out if you somehow are still logged in
Clear all cache, site data, etc.
Hard refresh Beehaw using CTRL+F5
Log back in.

If you still have issues, write to us at support@beehaw.org

To be clear : We have not been hacked as far as we know, we were completely unaffected. This was done preemptively.

Oh yeah, in case, you haven't, this is a good opportunity and reminder to follow us on Mastodon as the communication line was still up despite Beehaw being down : https://hachyderm.io/@beehaw

you are viewing a single comment's thread
view the rest of the comments

[-] comicallycluttered@beehaw.org 3 points 1 year ago

Agree with everyone else. Thanks for shutting it down.

I'll most likely do it anyway, but do you think password changes are necessary at this point?

[-] Lionir@beehaw.org 3 points 1 year ago

I don't think this is necessary.

We had no messages on our database that had the vulnerability though some were federated from blahaj in the aftermath. The JWT, which is your session token, was changed as well so it seems very unlikely to me that this needs to be changed. There's no reason to believe the attack could've given access to passwords.

[-] abhibeckert@beehaw.org 5 points 1 year ago* (last edited 1 year ago)

I don’t think this is necessary.

I'd add that it's basically useless. From what I've seen, resetting your password doesn't even invalidate previously issued JWT tokens, which would be the only reason to do it. But of course, you've already reset them all and so has lemmy.world.