submitted 1 week ago by BB84@mander.xyz to c/mander@mander.xyz

14 comments fedilink hide all child comments

Server is slow for me in the past few days. Anyone else feeling the same?

you are viewing a single comment's thread
view the rest of the comments

[-] Sal@mander.xyz 4 points 1 week ago

So far, I have been able to 'control' the CPU use by setting limits to the process that pulls stuff from the database (pool size, CPU, memory).

This does release some of the CPU for other tasks, but I think that that what creates the lag might actually be the clogged database queries. So, constraining those resources might not solve the lag problem.

[-] acockworkorange@mander.xyz 3 points 1 week ago

Did you upgrade Lemmy on the 13th or something? Could be a new bug.

Also, did site traffic increase at all? Maybe there's an AI scraper messing up.

[-] Sal@mander.xyz 7 points 1 week ago

I did not update or change anything in the past few days.

But, now that you mentioned an AI scrapper I looked into the logs and noticed some heavy requests to the API from a specific IP.

Requests look consistent to scraping - just consistently and continuously issuing GET requests to different API endpoints.

XX.XX.XX.XX - - [14/Oct/2025:21:28:54 +0000] "GET /api/v3/community/list?limit=20&sort=TopAll HTTP/2.0" 403 107 "-" "Mlmym"

I have started denying their requests and it is the first thing that seems to have actually helped!

I don't want to speak too early but I think you may have identified the cause. Thanks!

[-] Sal@mander.xyz 5 points 1 week ago

I just realized that it is not a 'scraper', the requests came from the server that I am using to provide an interface to the site as an Onion site. The amount of requests was suspiciously high so maybe a bot is scraping through Tor. I will leave it off for a few days and see if I can turn it back on later.

[-] acockworkorange@mander.xyz 3 points 1 week ago* (last edited 1 week ago)

That's a level of sophistication I hadn't seen - using Tor to scrape. Hopefully they give up soon. I'm glad we got some respite. Thank you!

[-] admiralpatrick@lemmy.world 2 points 1 week ago

Check your web access logs for these 3 IPs:

134.19.178.167
213.152.162.5
134.19.179.211

And see if they're making repeated un-scoped (no page paramater) requests to /api/v3/comment/list. If they are, block them in your firewall.

Those used to hit my instance constantly with requests like /api/v3/comment/list?sort=Old&page=16514 (yes, page 16,414). When I blocked those IPs making those requests, problem solved.

[-] Sal@mander.xyz 2 points 1 week ago

Thanks!

I don't see those specific IPs, nor 16514. But now I see what scrapers tend to look like in the logs :)

I am now pretty sure that the cause was scraper-like activity coming from the Mlmym front-end that I am serving over an onion site. I am not sure if it randomly started mis-behaving or if a tor scraper was using it.

After blocking this, federation was restored, performance increased, and CPU use came down:

this post was submitted on 14 Oct 2025

13 points (100.0% liked)

Mander

527 readers

1 users here now

founded 3 years ago

MODERATORS

Sal@mander.xyz