submitted 1 week ago by BB84@mander.xyz to c/mander@mander.xyz

14 comments fedilink hide all child comments

Server is slow for me in the past few days. Anyone else feeling the same?

top 14 comments

sorted by: hot top controversial new old

[-] Sal@mander.xyz 8 points 1 week ago

I can see that since October 13 the CPU use spiked to 100%...

I just reset the server to check if that would fix it quickly. It don't think it did.

I can see the processes that are taking up a lot of CPU (parallell database queries it seems), but I still don't understand what is causing these and why they have spiked since Oct 13. I will need to investigate.

[-] Sal@mander.xyz 4 points 1 week ago

So far, I have been able to 'control' the CPU use by setting limits to the process that pulls stuff from the database (pool size, CPU, memory).

This does release some of the CPU for other tasks, but I think that that what creates the lag might actually be the clogged database queries. So, constraining those resources might not solve the lag problem.

[-] acockworkorange@mander.xyz 3 points 1 week ago

Did you upgrade Lemmy on the 13th or something? Could be a new bug.

Also, did site traffic increase at all? Maybe there's an AI scraper messing up.

[-] Sal@mander.xyz 7 points 1 week ago

I did not update or change anything in the past few days.

But, now that you mentioned an AI scrapper I looked into the logs and noticed some heavy requests to the API from a specific IP.

Requests look consistent to scraping - just consistently and continuously issuing GET requests to different API endpoints.

XX.XX.XX.XX - - [14/Oct/2025:21:28:54 +0000] "GET /api/v3/community/list?limit=20&sort=TopAll HTTP/2.0" 403 107 "-" "Mlmym"

I have started denying their requests and it is the first thing that seems to have actually helped!

I don't want to speak too early but I think you may have identified the cause. Thanks!

[-] Sal@mander.xyz 5 points 1 week ago

I just realized that it is not a 'scraper', the requests came from the server that I am using to provide an interface to the site as an Onion site. The amount of requests was suspiciously high so maybe a bot is scraping through Tor. I will leave it off for a few days and see if I can turn it back on later.

[-] acockworkorange@mander.xyz 3 points 1 week ago* (last edited 1 week ago)

That's a level of sophistication I hadn't seen - using Tor to scrape. Hopefully they give up soon. I'm glad we got some respite. Thank you!

[-] admiralpatrick@lemmy.world 2 points 1 week ago

Check your web access logs for these 3 IPs:

134.19.178.167
213.152.162.5
134.19.179.211

And see if they're making repeated un-scoped (no page paramater) requests to /api/v3/comment/list. If they are, block them in your firewall.

Those used to hit my instance constantly with requests like /api/v3/comment/list?sort=Old&page=16514 (yes, page 16,414). When I blocked those IPs making those requests, problem solved.

[-] Sal@mander.xyz 2 points 1 week ago

Thanks!

I don't see those specific IPs, nor 16514. But now I see what scrapers tend to look like in the logs :)

I am now pretty sure that the cause was scraper-like activity coming from the Mlmym front-end that I am serving over an onion site. I am not sure if it randomly started mis-behaving or if a tor scraper was using it.

After blocking this, federation was restored, performance increased, and CPU use came down:

[-] Bonus@mander.xyz 1 points 1 week ago* (last edited 1 week ago)

~~It's loading way faster than it was this morning.~~

and, I may have spoken too soon.

Earlier today, it was hanging for some seconds with every click. Seemed faster for a moment but still laggy.

[-] Sal@mander.xyz 4 points 1 week ago

Hmm, thanks!

I just made another change. I am not sure why the database suddenly started taking up so many resources, so I have reduced the amount of resources available to it. It did reduce CPU use to 80% as opposed to 100%, but it is still somewhat laggy for me.

[-] Bonus@mander.xyz 2 points 1 week ago* (last edited 1 week ago)

Awesome, best of luck to you.

BTW, this brought me here today and had me noticing how slow it was loading:

Is there currently a fedaration issue between piefed.social and mander.xyz or am I doing something wrong?

https://piefed.social/post/1369461#comment_8420089

[-] Sal@mander.xyz 3 points 1 week ago

Thanks! It seems like the lag may have been due to a scraper

[-] Fourth@mander.xyz 2 points 1 week ago

Ty for your efforts

[-] Sal@mander.xyz 2 points 1 week ago

No problem! Happy to contribute

this post was submitted on 14 Oct 2025

13 points (100.0% liked)

Mander

527 readers

1 users here now

founded 3 years ago

MODERATORS

Sal@mander.xyz