147
Are all these thousands of lemmy servers useless?
(lemmy.world)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 12 Jun 2023
147 points (100.0% liked)
Selfhosted
40767 readers
488 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 2 years ago
MODERATORS
every instance is sharing in the traffic to browse the fediverse. Not one service is responsible for serving content, you (the instance admin) are only serving for your members.
The downside of this is there is a huge amount of replicated data stored everywhere. Content of popular communities will be scraped by and stored on many many servers, filling up servers and increasing storage and bandwith bills for all those servers
I'm not sure your second paragraph is correct. First of all, it's "just in time" so will only be replicated if somebody on that instance is following it. But more importantly, I read a statement from a server owner somewhere that the software purges older content regularly (and refetches is "just in time" when somebody tries to view the old content) to keep storage size down.
If this is the case, then wouldn't his fitst paragraph be incorrect also? Because if it is "just in time" with quick purging, the main server still has to constantly serve the instance server the content. It would only be beneficial if many instance users are trying to view the exact same content at around the same time (so for the "massive" communities maybe?)
From my understanding you are correct. Each instance is responsible for serving all of the content of the communities created on it. So many small instances with a smaller amount of communities = good, a few huge instances with lots of communities = bad.
The purging of older content is good news I didn’t know that.
Please elaborate, how is "every instance is sharing in the traffic to browse the fediverse". I didn't find it nor in AP standards, nor in activitypub_federation lib docs. If there is some mechanisms of balancing inside the lemmy's code, would you mind pointing it for me?
Looking into the database, it contains many thousands of posts. I’m assuming this is stored in the local db for serving it to instance members. So when you open a post from instance B on instance A, A fetches post-data from B, stores it in A database, then serve the content from db A to the browser
Yes, you are right. If this instance has members. A server will actively fetch "foreign" content and cache when this instance's user asks. But aside of top 10 servers, there is no profit of having more until they have a couple of dozens of users. If any server would have been able to "delegate" request handling to less busy servers, it will be a solution for this uneven load.
The replication isn't all that bad. Images stick around in their local instance, the federated data is all JSON payloads and metadata. Yes it will pile up over time, but only instances with hundreds of users and thousands of indexed communities are at risk of massive storage needs.