Huge props for being one of the few major instances to preemptively shut down!
shutting down the server early was best. the nature of open source software is what allows these incidents to be mitigated as quickly as they are. thanks a lot to you guys, and to all of the team at Lemmy who worked to resolve this.
heroes <3
Glad it's back up. I went outside. It was hot af and boring.
This is why I am on Beehaw. The Admins really care about the Instance and the content on it.
That's why I want to bring attention to the fact, that U can support them. https://opencollective.com/beehaw
I am not a Admin, Mod or anything else. I just really like Beehaw and support them. And you should too.
Thank you for shutting down rather than "wait and see"! It was the right choice.
The shutdown is a good call given the circumstances.
An idea of less-radical preventive action is placing the instance in read-only mode, either as a Lemmy feature, or through reverse proxy settings (eg reply 503 for any POST/PUT/DELETE request). But that'd require some development and/or preparation.
Doing that on the reserve proxy side would block any user-submitted content and more (logins, searches, ...). This would hopefully be efficient at blocking many attack vectors, while still keeping the instance partially online, even if that's a degraded mode.
Would read-only mode help with XSS exploits though, like this particular one? Since the "damage was already done" by the time anybody noticed, wouldn't putting the site in read-only mode still have kept serving up the XSS payload? It'd stop "infected" people from making any state mutations on Lemmy, but eg. data exliftration would still happen
Note that if this were a Lemmy feature, if we had been infected, an admin could've gotten hacked and as a result, disabled that feature. I'm not really sure what can be done to make Beehaw foolproof. That said, the UI has since been hardened by CSP headers so this type of attack should no longer be possible.
Thanks for the swift pre-emptive action
huge Ws, excellent work
also, thanks for the Mastodon link, i wasn't sure where to check on beehaw status during the outage
morning thought: I've definitely joined the right instance. (also the start from the assumption of good faith guidelines linked to in Gaywallet's recent post)
Awesome response, and a great succinct postmortem. Thanks for doing what you do!
Thank you for all you do, from what I was hearing I was in no way expecting you to have the site back up within 12 hours. Many kudos.
12:30AM EST: I make announcements to tell people about this
I think it'd be beneficial to have more backup lines of communication for announcements than just Mastodon.
We have Discord and Matrix channels as well. Do you have anything to suggest?
Something like status-page is always nice. I haven't used it but it looks like https://cachethq.io/ could be a decent fit as well.
Content-Security-Policy will really help save your ~~bacon~~ beans and protect against XSS. Hopefully the Lemmy devs can apply a super strict policy to help. IMHO it's a must for any site with user generated content.
Awesome work sidestepping the hack.
Your work is greatly appreciated! Also happy to know that you got some sleep, very important for the process ☺
Good work.
Have a non custom beer 🍻
Great job! Being preemptive in a case like this is very good! Thanks for all your work!
Welcome Back 🤗
Was the hack related to the lasted patch update?
I'm not sure what you're asking.
I don’t believe it was related to anything specifically added or removed in a recent patch, I think it was an existing exploit that just hadn’t been noticed.
No it was related to custom emojis.
Agree with everyone else. Thanks for shutting it down.
I'll most likely do it anyway, but do you think password changes are necessary at this point?
I don't think this is necessary.
We had no messages on our database that had the vulnerability though some were federated from blahaj in the aftermath. The JWT, which is your session token, was changed as well so it seems very unlikely to me that this needs to be changed. There's no reason to believe the attack could've given access to passwords.
I don’t think this is necessary.
I'd add that it's basically useless. From what I've seen, resetting your password doesn't even invalidate previously issued JWT tokens, which would be the only reason to do it. But of course, you've already reset them all and so has lemmy.world.
A password reset probably should invalidate all previous JWT tokens.
We had no messages on our database that had the vulnerability
This is interesting. I actually commented about the use of emojis/emotes a couple days ago on a post on !foss@beehaw.org made by a federated user from lemmy.one, that has since been removed (😕), but I still have the bookmarked comment in which I copied the raw embed for the remote emote image in the federated comment I was responding to.
Do I understand it correctly, that the latest fixes to stop the code injection, will still allow remote image embedding, so something like an "emote picker extension to embed animated GIFs from a remote server and/or remote instance's emoji list" would still be doable and wouldn't pose any risk?
Or would such picker still have to include measures to prevent offering embeds with potential exploits?
Remote image embedding is not the issue, remote custom emojis would not have been an issue either. The issue, from my understanding, is that the way local emojis are rendered allowed for an XSS exploit.
You can look at the PR which fixed this issue if you have a better understanding of these things than me : https://github.com/LemmyNet/lemmy-ui/pull/1897/
I believe such a picker would be fine.
I see, so the prior emoji handling rendered content directly from the comment, instead of making sure it was strictly what was defined for the local emoji; that was a weird choice. Now they've also added a sanitizer wrapper to all of it in: https://github.com/LemmyNet/lemmy-ui/pull/1906
I guess the only downside of a picker that used the non-emoji image renderer, would be the loss of emoji CSS formatting.
From what I can tell the whole point to the css class/formatting was controlling the size of the emojis. Depending on where they came from, I could see some being of random size and shape. Admins might not have the time or know-how to shrink them down, so css seems like a reasonable compromise as long as the files aren't huge.
I'm kind of bothered that the only fix seems to be on the frontend. Unfortunately, I haven't been able to stick with Rust long enough to take a reasonable crack at figuring out how to help on the backend. Input and output sanitization should ideally be handled in both places.
Support and meta community for Beehaw. Ask your questions about the community, technical issues, and other such things here.
A brief FAQ for lurkers and new users can be found here.
Our September 2024 financial update is here.
For a refresher on our philosophy, see also What is Beehaw?, The spirit of the rules, and Beehaw is a Community
This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.
