2264
submitted 2 years ago* (last edited 2 years ago) by ruud@lemmy.world to c/lemmyworld@lemmy.world

Status update July 4th

Just wanted to let you know where we are with Lemmy.world.

Issues

As you might have noticed, things still won't work as desired.. we see several issues:

Performance

  • Loading is mostly OK, but sometimes things take forever
  • We (and you) see many 502 errors, resulting in empty pages etc.
  • System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

Bugs

  • Replying to a DM doesn't seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
  • 2FA seems to be a problem for many people. It doesn't always work as expected.

Troubleshooting

We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @nutomic@lemmy.ml is also helping with current issues.

So, all is not yet running smoothly as we hoped, but with all this help we'll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!

(page 3) 50 comments
sorted by: hot top controversial new old
[-] EpicMuellsTime@lemmy.world 12 points 2 years ago

Can anyone give me a link to donate to help out? Ever since discovering lemmy I hardly use reddit and would like to contribute to this community more!

[-] Gold_ghost057@lemmy.world 12 points 2 years ago

Yep! Donating to both their Open Collective page for Mastodon.World (didn't grab that link by accident), as well as their Patreon page for Mastodon.World works!

They have a note up on Patreon that donations there also contribute to World.Lemmy, as well as another Lemmy instance!

Here's the link to their Patreon; https://patreon.com/mastodonworld

load more comments (1 replies)
[-] imaqtpie@sh.itjust.works 12 points 2 years ago* (last edited 2 years ago)

Thanks for the update, you're leading the way for Lemmy.

I was prepared for servers crashing and lagging, but you're keeping things moving thus far, despite receiving 85k new users in a month. Heroic stuff.

[-] el_cordoba@lemmy.world 12 points 2 years ago

I am sure the stress is unreal, but It is appreciated. You all stepped up when everyone was stepping down (or trying to sellout). I am hopeful this will turn into something great.

[-] GlintingKingfisher@lemmy.world 12 points 2 years ago

Thanks for the transparent update!

[-] ghariksforge@lemmy.world 11 points 2 years ago

Thanks for the transparency and keep up the good work!

[-] ben914@lemmy.world 11 points 2 years ago* (last edited 2 years ago)

Thanks for the transparency, and communication. I think it's always better when the userbase is able to understand what is going on rather than being left in the dark wondering what is going wrong. Keep up the good work, but also be sure you guys get enough rest, and take care of yourselves too.

[-] thatguy_ie@lemmy.world 11 points 2 years ago

Thank you for the transparency @ruud@lemmy.world It is rare for platforms to scale this quickly so issues like this are inevitable. Good luck for the troubleshooting!

[-] WildfireFailure@programming.dev 10 points 2 years ago

What sort of container orchestration and database tools are you using? Maybe you need some sys admin help?

[-] sykccc@lemmy.world 10 points 2 years ago

Y’all doing amazing things keeping us going πŸ”₯

[-] RichardButt89@lemmy.world 10 points 2 years ago

This has been such a refreshing change from reddit! I'm willing to deal with occasional hiccups, and I know it will only get better with time. Keep it up!

[-] scaredoftrumpwinning@lemmy.world 10 points 2 years ago

Thanks for keeping us apprised. Hopefully you will find the resource leak.

[-] QuentinCallaghan@sopuli.xyz 9 points 2 years ago

around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

25 GB of RAM????? Looks like your instance is the edge case with Lemmy's scalability, when there's really big amount of active users. Looks like some problems won't fade with really high-grade hardware. But I have to appreciate, that's one gargantuan job!

load more comments (2 replies)
[-] JeanMiaouss@lemmy.world 9 points 2 years ago

Thanks for all the work you, and all the others do to welcome so many new users, and thank you for keeping us posted on your efforts to smooth things over.

From what I see the general sentiment is very positive and understanding, and it feels special to be part of a community that's more upbeat than what you see in most online places.

[-] RoundSparrow@lemm.ee 9 points 2 years ago* (last edited 2 years ago)

I expect it is federation outbound activity growing in memory usage, it's all held in RAM for every post/comment/like going out to subscribed servers. !lemmyperformance@lemmy.ml is a community for the focus on scaling Lemmy.

load more comments (1 replies)
[-] mtnwolf@lemmy.world 9 points 2 years ago

I totally appreciate all of your efforts! Thank you for being a pioneer.

[-] Kosh420@lemmy.world 9 points 2 years ago

Thank you to you and everyone behind the scenes working hard to make Lemmy a great place. I am turkey enjoying the experience even with the bugs. Brings back a ton of nostalgia with modern mixed in.

[-] smoll_pp_operator@vlemmy.net 8 points 2 years ago

Yall are navigating waters that you probably have little professional experience with. Keep at it and don't get discouraged.

[-] WigglingWalrus@lemmy.world 8 points 2 years ago

Appreciate the update, best of luck.

[-] RaincoatsGeorge@lemmy.world 8 points 2 years ago

Its nice to have a place where the admins and developers aren't trying to scheme and manipulate the community to get a big payout. Thanks for all that you guys do. Curating this community is no doubt selfless hard work but we need a place like this free from the influence or potential take over by bad actors and malignant narcissists.

[-] YellowtoOrange@lemmy.world 8 points 2 years ago

As much as I hope that Lemmy grows, I wouldn't want lemmy.world to suddenly get 1 million signups and multiply your headaches.

Perhaps we need more well run servers by other people to take the load.

[-] Eh_I@lemmy.world 8 points 2 years ago

I can come whine in the discord if that'll help.

[-] LunchEnjoyer@lemmy.world 8 points 2 years ago

Thanks for making this possible in the first place 🀘

[-] brewery@lemmy.world 8 points 2 years ago

Amazing work team. I am already seeing improvements. Hope you are not killing yourselves though, I'm sure everyone realises how difficult it is and that it will take time to fix. We're here for the long haul! Thanks again

[-] Aimhere@lemmy.world 8 points 2 years ago

Thank you for everything you do. With any luck, you'll get all the support you need, and have it running smooth as silk.

[-] WubbyGeth@lemmy.world 8 points 2 years ago

Do you need any spare server hardware? @ruud@lemmy.world I would be happy to donate some!

[-] ruud@lemmy.world 10 points 2 years ago

No thanks, it's all at Hetzner, and thanks to all donations we can extend when needed!

[-] footlover7777@lemmy.world 7 points 2 years ago

Thanks for putting in all the hard work. Especially on a holiday. We appreciate you guys!

load more comments (1 replies)
[-] SoulKeeper@lemmy.world 7 points 2 years ago* (last edited 2 years ago)

Sometimes when I post a comment (without any image) it takes forever to post and sometimes it keeps spinning and I need to cancel and post again. My internet connection is fine (100 Mbps) and it works flawlessly on other websites. Is it happening due to overloading on the lemmy server?

load more comments (3 replies)
[-] ABeeinSpace@lemmy.world 7 points 2 years ago

Thanks for the update!

load more comments
view more: β€Ή prev next β€Ί
this post was submitted on 04 Jul 2023
2264 points (100.0% liked)

Lemmy.World Announcements

30793 readers
3 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world/

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS