2883
submitted 1 year ago* (last edited 1 year ago) by ruud@lemmy.world to c/lemmyworld@lemmy.world

While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

  • There is an vulnerability which was exploited
  • Several people had their JWT cookies leaked, including at least one admin
  • Attackers started changing site settings and posting fake announcements etc

Our mitigations:

  • We removed the vulnerability
  • Deleted all comments and private messages that contained the exploit
  • Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been 'stolen' and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

you are viewing a single comment's thread
view the rest of the comments
[-] Nugget@lemmy.world 22 points 1 year ago

I think this is a strong reminder: We shouldn't put all our eggs in one basket. This will happen again. Unlike Reddit, we don't need to concentrate all communities on one instance. We should all make an effort to spread out. Some other general use instances are:

Again, for those new, you can post content to any of these instances and interact with content from other instances at the same time, just like you can send an email from your Gmail account to your ProtonMail account.

[-] SQL_InjectMe@partizle.com 11 points 1 year ago

This was an exploit in lemmy software and affected multiple instances though

[-] dragontamer@lemmy.world 3 points 1 year ago

The malicious Javascript was in a post however, and that post spread to all Lemmies that saw the post.

I'm pretty sure all web browsers who saw that post passed the JWT cookie to the hacker. The hacker was looking for admin accounts and got the Lemmy.world admin first.

[-] DreadTowel@lemmy.world 3 points 1 year ago

Would be great to have identity independent from any particular servers...

[-] 0x4E4F@sh.itjust.works 2 points 1 year ago* (last edited 1 year ago)

That is currently not possible in the fediverse. Might be some day.

[-] DreadTowel@lemmy.world 2 points 1 year ago

Yeah, I know. It is possible to implement it though. Posts and comments are cached in every federated server, the only thing you need is attribution to the author. Your key/certificate could be your identity.

[-] kakes@sh.itjust.works 2 points 1 year ago

I've thought about this, and had come to the conclusion that it would be too difficult for users to manage their private key. (Look at the flustercuck we've seen in cryptocurrency)

I wonder though, if we could choose to either manage our own private key OR let an instance manage it for us. This way, users could choose to give full control to an instance (as it is now) or to keep control themselves for extra security.

User data and public keys would still need to be stored (presumably on an instance server, though I wouldn't be against separating this duty into a separate server which may or may not be hosted concurrently with an instance). But at least this way, we would have the option to keep control over our accounts even if the instance is hacked.

[-] DreadTowel@lemmy.world 1 points 1 year ago

If I had to propose something, Iโ€™d suggest writing a spec and implementing an API for Lemmy servers that would allow one to submit signed actions, like posts, comments, likes, display name, etc. Then people will write clients that will allow generation of certificates to be used to sign those messages + add the ability to export/back up the key to be used by other apps too.

I think people who actually understood crypto used it correctly. It's just that most people were there to speculate and gamble. And, tbh, I think crypto is here to stay.

Yup, I'd support an option to store the key on a server. Or just use an account tied to a server, just like now. Currently, servers sign all messages to other instances with their key, so all messages are tied to the identity of the instance!

[-] kakes@sh.itjust.works 2 points 1 year ago

100%. Also, while we're at it, I just want to throw it out there that the front-end should also be split out imo. Let people mix and match backend/user/frontend according to a central api specification, so the best options for each can rise to the top, rather than relying on a single monolithic codebase.

[-] DreadTowel@lemmy.world 3 points 1 year ago
[-] kakes@sh.itjust.works 2 points 1 year ago

Ahh well never mind then! For some reason I thought it was all together - maybe I was thinking about the docker-compose or something. Thanks for correcting!

[-] 0x4E4F@sh.itjust.works 1 points 1 year ago

It's... slightly more complicated than that, but it should be possible.

This was one of my early fears... that an instance might get offline, attacked, whatever, and that all your user data might be lost. It appears that a feature like this might be inevitable. Or at least a feature that would let a user to have at least 1 copy of his account on another instance.

There is also the DB and storage growth to consider if something like this is implemeneted. If all instances hold all data from every other instance, than that might turn into a shit show pretty soon. You can get away with a certain number of users on only one server, but after a certain threshold, you have to scale... and that costs a lot of money, which is a no go for smaller instances and they will probably shut down if they're forced to hold data for every user account in the fediverse.

[-] DreadTowel@lemmy.world 1 points 1 year ago

If I had to propose something, I'd suggest writing a spec and implementing an API for Lemmy servers that would allow one to submit signed actions, like posts, comments, likes, display name, etc. Then people will write clients that will allow generation of certificates to be used to sign those messages + add the ability to export/back up the key to be used by other apps too.

If storage space ever becomes an issue, you could potentially shard communities. I'm more worried about the network traffic. I already suggested a routing algorithm that would spread the load between all federated instances and would scale like O(log(N)). There was some interest, but it would be a long term project. There's still a lot of performance that can be gained by simple optimisations.

[-] 0x4E4F@sh.itjust.works 2 points 1 year ago* (last edited 1 year ago)

Yeah, there are more pressing issues at hand, a lot of bugs, a lot of new features that need to be implemented ASAP. Like user following, Lemmy is still lacking that.

[-] steltek@lemm.ee 1 points 1 year ago

Accounts aren't just posts/comments/votes. You have the communities you're following, settings, and other bits of private state (remembering what you've already read). You'd be in charge of preserving and lugging this data around.

[-] DreadTowel@lemmy.world 1 points 1 year ago

Either store that data yourself or rely on your servers to store it and distribute it, just like they do today.

[-] Kissaki@feddit.de 2 points 1 year ago

Register on 5 so you're vulnerable on 5, got it.

[-] Caminsky@lemmy.world 1 points 1 year ago

Is it possible to replicate to another community as a back up? Kind of a Master-slave DB?

[-] solarbabies@lemmy.world 1 points 1 year ago

Did any of the other instances suffer this vulnerability? Or are they all running different code? i.e. different repositories/implementations of Lemmy

this post was submitted on 10 Jul 2023
2883 points (100.0% liked)

Lemmy.World Announcements

29028 readers
3 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news ๐Ÿ˜

Outages ๐Ÿ”ฅ

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations ๐Ÿ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 1 year ago
MODERATORS