633
submitted 1 week ago* (last edited 1 week ago) by SmokeyDope@piefed.social to c/selfhosted@lemmy.world

I got into the self-hosting scene this year when I wanted to start up my own website run on old recycled thinkpad. A lot of time was spent learning about ufw, reverse proxies, header security hardening, fail2ban.

Despite all that I still had a problem with bots knocking on my ports spamming my logs. I tried some hackery getting fail2ban to read caddy logs but that didnt work for me. I nearly considered giving up and going with cloudflare like half the internet does. But my stubbornness for open source self hosting and the recent cloudflare outages this year have encouraged trying alternatives.

Coinciding with that has been an increase in exposure to seeing this thing in the places I frequent like codeberg. This is Anubis, a proxy type firewall that forces the browser client to do a proof-of-work security check and some other nice clever things to stop bots from knocking. I got interested and started thinking about beefing up security.

I'm here to tell you to try it if you have a public facing site and want to break away from cloudflare It was VERY easy to install and configure with caddyfile on a debian distro with systemctl. In an hour its filtered multiple bots and so far it seems the knocks have slowed down.

https://anubis.techaro.lol/

My botspam woes have seemingly been seriously mitigated if not completely eradicated. I'm very happy with tonights little security upgrade project that took no more than an hour of my time to install and read through documentation. Current chain is caddy reverse proxy -> points to Anubis -> points to services

Good place to start for install is here

https://anubis.techaro.lol/docs/admin/native-install/

top 50 comments
sorted by: hot top controversial new old
[-] termaxima@slrpnk.net 2 points 6 days ago

I am very annoyed that I have to enable cloudflare's JavaScript on so many websites, I would much prefer if more of them used Anubis so I didn't have third-party JavaScript running as often.

( coming from an annoying user who tries to enable the fewest things possible in NoScript )

[-] non_burglar@lemmy.world 189 points 1 week ago

Anubis is an elegant solution to the ai bot scraper issue, I just wish the solution to everything wasn't just spending compute everywhere. In a world where we need to rethink our energy consumption and generation, even on clients, this is a stupid use of computing power.

[-] Dojan@pawb.social 116 points 1 week ago* (last edited 1 week ago)

It also doesn’t function without JavaScript. If you’re security or privacy conscious chances are not zero that you have JS disabled, in which case this presents a roadblock.

On the flip side of things, if you are a creator and you’d prefer to not make use of JS (there’s dozens of us) then forcing people to go through a JS “security check” feels kind of shit. The alternative is to just take the hammering, and that feels just as bad.

No hate on Anubis. Quite the opposite, really. It just sucks that we need it.

[-] SmokeyDope@piefed.social 63 points 1 week ago* (last edited 1 week ago)

Theres a compute option that doesnt require javascript. The responsibility lays on site owners to properly configure IMO, though you can make the argument its not default I guess.

https://anubis.techaro.lol/docs/admin/configuration/challenges/metarefresh

From docs on Meta Refresh Method

Meta Refresh (No JavaScript)

The metarefresh challenge sends a browser a much simpler challenge that makes it refresh the page after a set period of time. This enables clients to pass challenges without executing JavaScript.

To use it in your Anubis configuration:

# Generic catchall rule
- name: generic-browser
  user_agent_regex: >-
    Mozilla|Opera
  action: CHALLENGE
  challenge:
    difficulty: 1 # Number of seconds to wait before refreshing the page
    algorithm: metarefresh # Specify a non-JS challenge method

This is not enabled by default while this method is tested and its false positive rate is ascertained. Many modern scrapers use headless Google Chrome, so this will have a much higher false positive rate.

[-] z3rOR0ne@lemmy.ml 15 points 1 week ago

Yeah I actually use the noscript extension and i refuse to just whitelist certain sites unless I'm very certain I trust them.

I run into Anubis checks all the time and while I appreciate the software, having to consistently temporarily whitelist these sites does get cumbersome at times. I hope they make this noJS implementation the default soon.

load more comments (2 replies)
load more comments (1 replies)
[-] natecox@programming.dev 22 points 1 week ago

I feel comfortable hating on Anubis for this. The compute cost per validation is vanishingly small to someone with the existing budget to run a cloud scraping farm, it’s just another cost of doing business.

The cost to actual users though, particularly to lower income segments who may not have compute power to spare, is annoyingly large. There are plenty of complaints out there about Anubis being painfully slow on old or underpowered devices.

Some of us do actually prefer to use the internet minus JS, too.

Plus the minor irritation of having anime catgirls suddenly be a part of my daily browsing.

load more comments (6 replies)
[-] cecilkorik@piefed.ca 13 points 1 week ago

if you are a creator and you’d prefer to not make use of JS (there’s dozens of us) then forcing people to go through a JS “security check” feels kind of shit. The alternative is to just take the hammering, and that feels just as bad.

I'm with you here. I come from an older time on the Internet. I'm not much of a creator, but I do have websites, and unlike many self-hosters I think, in the spirit of the internet, they should be open to the public as a matter of principle, not cowering away for my own private use behind some encrypted VPN. I want it to be shared. Sometimes that means taking a hammering. It's fine. It's nothing that's going to end the world if it goes down or goes away, and I try not to make a habit of being so irritating that anyone would have much legitimate reason to target me.

I don't like any of these sort of protections that put the burden onto legitimate users. I get that's the reality we live in, but I reject that reality, and substitute my own. I understand that some people need to be able to block that sort of traffic to be able to limit and justify the very real costs of providing services for free on the Internet and Anubis does its job for that. But I'm not one of those people. It has yet to cost me a cent above what I have already decided to pay, and until it does, I have the freedom to adhere to my principles on this.

To paraphrase another great movie: Why should any legitimate user be inconvenienced when the bots are the ones who suck. I refuse to punish the wrong party.

load more comments (2 replies)
[-] cadekat@pawb.social 16 points 1 week ago

Scarcity is what powers this type of challenge: you have to prove you spent a certain amount of electricity in exchange for access to the site, and because electricity isn't free, this imposes a dollar cost on bots.

You could skip the detour through hashes/electricity and do something with a proof-of-stake cryptocurrency, and just pay for access. The site owner actually gets compensated instead of burning dead dinosaurs.

Obviously there are practical roadblocks to this today that a JavaScript proof-of-work challenge doesn't face, but longer term...

[-] daniskarma@lemmy.dbzer0.com 2 points 6 days ago

I think the issue is that many sites are too aggressive with it. Anubis can be configured to only ask for challenges if the site is under unusual load, for instance when a botnet it's actually ddosing the site. That's when it shines.

Making it constantly ask for challenges when the service is not under attack is just a massive waste of energy. And many sites just enable it constantly because they can defer bot pings from their logs that way. That's for instance what op is doing. It's just a big misunderstanding of the tool.

load more comments (5 replies)
load more comments (1 replies)
[-] sudo@programming.dev 55 points 1 week ago

I've repeatedly stated this before: Proof of Work bot-management is only Proof of Javascript bot-management. It is nothing to a headless browser to by-pass. Proof of JavaScript does work and will stop the vast majority of bot traffic. That's how Anubis actually works. You don't need to punish actual users by abusing their CPU. POW is a far higher cost on your actual users than the bots.

Last I checked Anubis has an JavaScript-less strategy called "Meta Refresh". It first serves you a blank HTML page with a <meta> tag instructing the browser to refresh and load the real page. I highly advise using the Meta Refresh strategy. It should be the default.

I'm glad someone is finally making an open source and self hostable bot management solution. And I don't give a shit about the cat-girls, nor should you. But Techaro admitted they had little idea what they were doing when they started and went for the "nuclear option". Fuck Proof of Work. It was a Dead On Arrival idea decades ago. Techaro should strip it from Anubis.

I haven't caught up with what's new with Anubis, but if they want to get stricter bot-management, they should check for actual graphics acceleration.

[-] SmokeyDope@piefed.social 38 points 1 week ago* (last edited 1 week ago)

Something that hasn't been mentioned much in discussions about Anubis is that it has a graded tier system of how sketchy a client is and changing the kind of challenge based on a a weighted priority system.

The default bot policies it comes with has it so squeaky clean regular clients are passed through, then only slightly weighted clients/IPs get the metarefresh, then its when you get to moderate-suspicion level that JavaScript Proof of Work kicks. The bot policy and weight triggers for these levels, challenge action, and duration of clients validity are all configurable.

It seems to me that the sites who heavy hand the proof of work for every client with validity that only last every 5 minutes are the ones who are giving Anubis a bad wrap. The default bot policy settings Anubis comes with dont trigger PoW on the regular Firefox android clients ive tried including hardened ironfox. meanwhile other sites show the finger wag every connection no matter what.

Its understandable why some choose strict policies but they give the impression this is the only way it should be done which Is overkill. I'm glad theres config options to mitigate impact normal user experience.

load more comments (1 replies)
[-] rtxn@lemmy.world 18 points 1 week ago* (last edited 1 week ago)

POW is a far higher cost on your actual users than the bots.

That sentence tells me that you either don't understand or consciously ignore the purpose of Anubis. It's not to punish the scrapers, or to block access to the website's content. It is to reduce the load on the web server when it is flooded by scraper requests. Bots running headless Chrome can easily solve the challenge, but every second a client is working on the challenge is a second that the web server doesn't have to waste CPU cycles on serving clankers.

POW is an inconvenience to users. The flood of scrapers is an existential threat to independent websites. And there is a simple fact that you conveniently ignored: it fucking works.

load more comments (11 replies)
load more comments (6 replies)
[-] daniskarma@lemmy.dbzer0.com 37 points 1 week ago* (last edited 1 week ago)

I don't think you have a usecase for Anubis.

Anubis is mainly aimed against bad AI scrappers and some ddos mitigation if you have a heavy service.

You are getting hit exactly the same, anubis doesn't put up a block list or anything. It just put itself in front of the service. The load on your server and the risk you take it's very similar anubis or not anubis here. Most bots are not AI scrappers they are just proving. So the hit on your server is the same.

What you want is to properly set up fail2ban or, even better, crowdsec. That would actually block and ban bots that try to prove your server.

If you are just self-hosting with Anubis the only thing you are doing is deriving the log noise towards Anubis logs and making your devices do a PoW every once in a while when you want to use your services.

Being honest I don't know what you are self hosting. But at least it's something that's going to get ddos or AI scrapped, there's not much point with Anubis.

Also Anubis is not a substitute for fail2ban or crowdsec. You need something to detect and ban brute force attacks. If not the attacker would only need to execute the anubis challenge get the token for the week and then they are free to attack your services as they like.

[-] natecox@programming.dev 36 points 1 week ago
[-] Cyberflunk@lemmy.world 16 points 1 week ago

thank you! this needed said.

  • This post is a bit critical of a small well-intentioned project, so I felt obliged to email the maintainer to discuss it before posting it online. I didn’t hear back.

i used to watch the dev on mastodon, they seemed pretty radicalized on killing AI, and anyone who uses it (kidding!!) i'm not even surprised you didn't hear back

great take on the software, and as far as i can tell, playwright still works/completes the unit of work. at scale anubis still seems to work if you have popular content, but does hasnt stopped me using claude code + virtual browsers

im not actively testing it though. im probably very wrong about a few things, but i know anubis isn't hindering my personal scraping, it does fuck up perplexity and chatgpt bots, which is fun to see.

good luck Blue team!

[-] kilgore_trout@feddit.it 16 points 1 week ago

the dev […] seemed pretty radicalized on killing Ai

As one should, to lead a similar project.

load more comments (2 replies)
[-] quick_snail@feddit.nl 29 points 1 week ago

Kinda sucks how it makes websites inaccessible to folks who have to disable JavaScript for security.

[-] poVoq@slrpnk.net 34 points 1 week ago

I kinda sucks how AI scrapers make websites inaccessible to everyone 🙄

load more comments (7 replies)
load more comments (4 replies)
[-] 0_o7@lemmy.dbzer0.com 29 points 1 week ago

I don't mind Anubis but the challenge page shouldn't really load an image. It's wasting extra bandwidth for nothing.

Just parse the challenge and move on.

[-] Allero@lemmy.today 23 points 1 week ago

Afaik, you can set it up not to have any image, or have any other one.

load more comments (2 replies)
[-] kilgore_trout@feddit.it 18 points 1 week ago* (last edited 1 week ago)

It's a palette of 10 colours. I would guess it uses an indexed colorspace, reducing the size to a minimum.
edit: 28 KB on disk

load more comments (5 replies)
[-] url@feddit.fr 23 points 1 week ago

Honestly im not a big fan of anubis . it fucks users with slow devices

https://lock.cmpxchg8b.com/anubis.html

[-] url@feddit.fr 14 points 1 week ago

Did i forgot to mention it doesnt work without js that i keep disabled

[-] smh@slrpnk.net 20 points 1 week ago

The creator is active on a professional slack I'm on and they're lovely and receptive to user feedback. Their tool is very popular in the online archives/cultural heritage scene (we combine small budgets and juicy, juicy data).

My site has enabled js-free screening when the site load is low, under the theory that if the site load is too high then no one's getting in anyway.

[-] panda_abyss@lemmy.ca 16 points 1 week ago

I like the quirky SPH character

[-] A_norny_mousse@feddit.org 16 points 1 week ago* (last edited 1 week ago)

At the time of commenting, this post is 8h old. I read all the top comments, many of them critical of Anubis.

I run a small website and don't have problems with bots. Of course I know what a DDOS is - maybe that's the only use case where something like Anubis would help, instead of the strictly server-side solution I deploy?

I use CrowdSec (it seems to work with caddy btw). It took a little setting up, but it does the job.
(I think it's quite similar to fail2ban in what it does, plus community-updated blocklists)

Am I missing something here? Why wouldn't that be enough? Why do I need to heckle my visitors?

Despite all that I still had a problem with bots knocking on my ports spamming my logs.

By the time Anubis gets to work, the knocking already happened so I don't really understand this argument.

If the system is set up to reject a certain type of requests, these are microsecond transactions of no (DDOS exception) harm.

load more comments (10 replies)
[-] sudoer777@lemmy.ml 15 points 1 week ago* (last edited 1 week ago)

I host my main server on my own hardware, and a VPN on Hetzner because my shitty ISP doesn't let me port forward. For the past year, bots were hitting my Forgejo instance hard. I forgot to disable registration and they generated hundreds of accounts with hundreds of repos with sketchy links, generating terrabytes of traffic from my VPS, costing me money in traffic. I disabled registration and deleted the spam, and bots still kept hitting my server for several months, which would cause memory leaks over time and crash it and consume CPU, and still costed me money with terrabytes of traffic per month. A few weeks ago, I put Anubis on the VPS. Now, zero bots hit my Forgejo instance and I don't pay for their traffic anymore. Problem solved.

[-] LOLseas@lemmy.zip 1 points 6 days ago

This is the first time I've ever seen it misspelled like that. It's 'terabyte/terabytes'. 1,024 GBs worth of data.

[-] sudoer777@lemmy.ml 3 points 6 days ago

Oops, although terabyte is 1000 GB, 1024 GiB is tebibyte

[-] LOLseas@lemmy.zip 1 points 5 days ago

Thanks friend. I only knew of the JEDEC terms, TIL.

[-] WorldsDumbestMan@lemmy.today 1 points 6 days ago

Nice ads people! Good job!

[-] helix@feddit.org 5 points 6 days ago

So you think techaro paid them?

[-] WorldsDumbestMan@lemmy.today 1 points 6 days ago

No clue, but it sounds so ad like...

load more comments (1 replies)
[-] TerHu@lemmy.dbzer0.com 14 points 1 week ago

yes, please be mindful when using cloudflare. with them you’re possibly inviting in a much much bigger problem

https://www.devever.net/~hl/cloudflare

load more comments (1 replies)
[-] drkt_@lemmy.dbzer0.com 13 points 1 week ago

Stop playing wack-a-mole with these fucking people and build TARPITS!

Make it HURT to crawl your site illegitimately.

load more comments
view more: next ›
this post was submitted on 28 Nov 2025
633 points (100.0% liked)

Selfhosted

53408 readers
201 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

  7. No low-effort posts. This is subjective and will largely be determined by the community member reports.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS