> In my experience the kinds of people who run this exact combination of circumstances also tend to be the kind of people that have a wide variance in the level of kindness they display to the authors of open source programs that happen to be in their way.
Love this. There’s been past discussion on HN re how OSS maintainers are treated, and this is such a wry sentence. I really appreciate the tone / attitude to the problem.
It's still very unclear to me how Anubis is supposed to work in the long term or even the medium term. The assumption is that scrappers, a.k.a. AI companies, a.k.a. those who hog all the RAM, are short of memory...?
The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute.
Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amount of crawling and that is the entire goal.
> requiring 1 second of compute before allowing a page load
After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike will need to spend the compute only once to get a valid session for themselves
Seems to be working so far though. Fwiw, lots of silly things stop bots until someone puts their mind (or tokens) to bypassing it, and then the blocking mechanism has to adapt. We'll see how it plays out, for now it's a lot better than begging big brother (cloudflare, google, or amazon captcha) for access, so I'm quite appreciative of what Techaro is doing
The issue is that scrapers are rotating their IP and essentially performing a DDoS attack. Anubis is part of a defense-in-depth solution. If scrapers reuse a cookie, traditional anti-abuse mechanisms will work.
Yep, on my aging phone it takes quite w noticable amount of time to go through. But on my more recent laptop wnd workstations, it's a quick flash of a manga lolita.
You could just spend 30 seconds googling instead of confidently asserting nonsense. There are plenty of people demonstrating that Anubis reduces bot traffic.
>There are plenty of people demonstrating that Anubis reduces bot traffic.
"Anubis reduces bot traffic" and "[the proof of work] is not stopping anyone" isn't contradictory, especially if anubis isn't being benchmarked against non-POW alternatives like IP/ASN bans, or tls fingerprinting.
>The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute.
Which doesn't hold. A 1s delay, despite being inperceptible is still a cost to human users. After all, 3600 x 1s delay adds up to an hour. Worse yet, the typical human visitor will always have to solve a challenge, because chances are it's the first time they're visiting your blog or whatever. Someone scraping reuse a valid session for at least a few more pages.
There's also the problem of time. It might be tempting to think the cost of a challenge is 1M cpu cycles or whatever, but the cost is better modeled as two parts, a time cost (for the human) and a cpu cost (for the computer). A scraper might have to pay the cpu cost, but not the time cost, because they can have their scraper solve challenges in their sleep. Unless the user is utterly destitute, their time is worth far more than whatever the compute cost is.
The assumption is that there will always be some population distribution in terms of effort expended as well as in terms of problems caused. At present it's the lowest effort actors that are causing the highest number of problems. So an 80% (or even just a 20%) solution largely solves the practical problem faced by site operators.
If against all odds it turns out that skiddies (or agents or AI billionaires or what have you) are willing to burn inordinate amounts of RAM and CPU just to scrape autogenerated cgit pages that are of approximately zero value to them to begin with then I guess we'll just have to cross that bridge when we come to it.
Is there a place where I can try out if my browser is compatible? Easier to find out now than when I'm trying to get work done and a million websites now have it deployed
On https://wasm-feature-detect.surma.technology it shows that I don't have 3 of all these features but I'm not sure if Anubis needs any of them to not kick me back to the pure JS solution
Which would apparently be bad because
> The WebAssembly that's shipped with this flow is ridiculously performant. This may mean you need to adjust the difficulty [upward, to avoid that bots solve it trivially when they support wasm, I assume this means]
combined with
> The wasm2js flow doesn't currently have a way to update the progress bar [so you have no clue about remaining time]
(wouldn't be the first time that I gave up on a page because it was stuck on 0 hashes per second)
It would be cool if the POW could be done ahead of time. That way I don't get stuck waiting while I'm working. Some type of credit/tokens that my browser could then spend.
That just creates another problem: if you're taking any measures to reduce tracking (ie. clearing cookies on shutdown or using temporary containers), this won't work. If anubis was being deployed on a site that a user visits often (eg. HN), the user might be convinced to whitelist it, but most anubis deployments are on random blogs or fediverse instances that I might not visit again in months. I'm certainly am not going to whitelist those sites, nor am I going to enable cookies wholesale just to avoid solving challenges.
Then we could implement an exchange, so that if you generate too much Anubis POW, you could exchange it with others.
We could link it to a site, you generate for HN, I for Reddit, but it so happens that you visited Reddit more and I HN, so we depleted our Anubis POW, so we could exchange some Reddit Anubis with some HN Anubis.
Anubis is fine if you want to block bots that are bothering you, but I never understood why they don't have robots.txt at the root level too. I happily obey that for my private crawler. I suppose you don't want to duplicate implementations. Fair enough. I guess I should add an Anubis detector so I can just blacklist because that person clearly doesn't want bots.
It's wild to me that PoW systems are how we sort the bot problem... the bummer is that all this "work" is just wasted cycles, at least in crypto there is a token you can sell.
Scraping would be costly in this world: scrapers would have to spend tokens in order to get the webpage.
But, in this world, the website owner would receive tokens that they can then use to do whatever they want, including paying for servers and bandwidth. This is the sense in which the cycles aren’t wasted: the website owner now has cash to spend.
Effectively, both scrapers and ordinary users would be paying for the privilege of getting website bytes.
This also solves the problem of having to wait for your phone to solve the challenge while you’re browsing: you can buy or mine some tokens ahead of time and pay them as soon as challenged. So can the scrapers, but because they’re accessing enormous numbers of pages it’s hopefully prohibitively expensive for them.
Yup was thinking the same: make honest people pay $0.00001 when they visit the site (in electricity/compute), have the challenge made so that only the website wins a tiny something. Bleed the bots dry.
> This makes Anubis challenges use a memory-hard proof of work function (argon2id) instead of just a CPU hard one. It also means that the "hey Claude vibeslop me a CUDA Anubis solver" route is on its way to being fundamentally dead.
I want to do this eventually, but it's hard to split things into the micro-tasks that would be required to make this work on Anubis. One of the ideas I'm throwing around is a world where Anubis helps fuzz old games to find timesaves in tool-assisted speedruns. It's harder than you think.
The goal isn't to eliminate scrapers, its to prevent a distributed scraping network from requesting 10000 pages a second each from 10000 different websites.
Yes, I also strongly suspect that this is only going to move more parts of scrapers onto consumer devices. The egress proxies are already there, why not use a little bit of the compute as well?
> In my experience the kinds of people who run this exact combination of circumstances also tend to be the kind of people that have a wide variance in the level of kindness they display to the authors of open source programs that happen to be in their way.
Love this. There’s been past discussion on HN re how OSS maintainers are treated, and this is such a wry sentence. I really appreciate the tone / attitude to the problem.
It's still very unclear to me how Anubis is supposed to work in the long term or even the medium term. The assumption is that scrappers, a.k.a. AI companies, a.k.a. those who hog all the RAM, are short of memory...?
The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute.
Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amount of crawling and that is the entire goal.
> requiring 1 second of compute before allowing a page load
After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike will need to spend the compute only once to get a valid session for themselves
Seems to be working so far though. Fwiw, lots of silly things stop bots until someone puts their mind (or tokens) to bypassing it, and then the blocking mechanism has to adapt. We'll see how it plays out, for now it's a lot better than begging big brother (cloudflare, google, or amazon captcha) for access, so I'm quite appreciative of what Techaro is doing
The issue is that scrapers are rotating their IP and essentially performing a DDoS attack. Anubis is part of a defense-in-depth solution. If scrapers reuse a cookie, traditional anti-abuse mechanisms will work.
It’s not seconds though, it’s in the order of tens of milliseconds of work, it’s faster to complete the challenge than most http requests will take.
last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)
This takes many minutes to solve on a cellphone, 9 milliseconds on a cheap VPS. It’s not stopping anyone.
Yep, on my aging phone it takes quite w noticable amount of time to go through. But on my more recent laptop wnd workstations, it's a quick flash of a manga lolita.
>It’s not stopping anyone.
You could just spend 30 seconds googling instead of confidently asserting nonsense. There are plenty of people demonstrating that Anubis reduces bot traffic.
>There are plenty of people demonstrating that Anubis reduces bot traffic.
"Anubis reduces bot traffic" and "[the proof of work] is not stopping anyone" isn't contradictory, especially if anubis isn't being benchmarked against non-POW alternatives like IP/ASN bans, or tls fingerprinting.
"it's not stopping anyone" and "it is stopping some people" are mutually exclusive statements.
You're selectively quoting the comment. It's clearly talking about the PoW aspect not being a serious impediment.
>The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute.
Which doesn't hold. A 1s delay, despite being inperceptible is still a cost to human users. After all, 3600 x 1s delay adds up to an hour. Worse yet, the typical human visitor will always have to solve a challenge, because chances are it's the first time they're visiting your blog or whatever. Someone scraping reuse a valid session for at least a few more pages.
There's also the problem of time. It might be tempting to think the cost of a challenge is 1M cpu cycles or whatever, but the cost is better modeled as two parts, a time cost (for the human) and a cpu cost (for the computer). A scraper might have to pay the cpu cost, but not the time cost, because they can have their scraper solve challenges in their sleep. Unless the user is utterly destitute, their time is worth far more than whatever the compute cost is.
The assumption is that there will always be some population distribution in terms of effort expended as well as in terms of problems caused. At present it's the lowest effort actors that are causing the highest number of problems. So an 80% (or even just a 20%) solution largely solves the practical problem faced by site operators.
If against all odds it turns out that skiddies (or agents or AI billionaires or what have you) are willing to burn inordinate amounts of RAM and CPU just to scrape autogenerated cgit pages that are of approximately zero value to them to begin with then I guess we'll just have to cross that bridge when we come to it.
Comparatively, this is what it's like getting all of this working across browser versions, platforms, and so many other things:
…<img not found/>…
I don’t know what this picture was supposed to be, but a 404 demonstrates your point perfectly well.
Refresh, the Anubis docs site regenerates its cookie secret on every deploy as part of the "holy shit did I break everything?" verification step.
Is there a place where I can try out if my browser is compatible? Easier to find out now than when I'm trying to get work done and a million websites now have it deployed
On https://wasm-feature-detect.surma.technology it shows that I don't have 3 of all these features but I'm not sure if Anubis needs any of them to not kick me back to the pure JS solution
Which would apparently be bad because
> The WebAssembly that's shipped with this flow is ridiculously performant. This may mean you need to adjust the difficulty [upward, to avoid that bots solve it trivially when they support wasm, I assume this means]
combined with
> The wasm2js flow doesn't currently have a way to update the progress bar [so you have no clue about remaining time]
(wouldn't be the first time that I gave up on a page because it was stuck on 0 hashes per second)
It would be cool if the POW could be done ahead of time. That way I don't get stuck waiting while I'm working. Some type of credit/tokens that my browser could then spend.
I'm working on this with a private prototype. I'm probably going to lean towards using a Service Worker (https://developer.mozilla.org/en-US/docs/Web/API/Service_Wor...) to renew challenges at a reduced difficulty. Stay tuned!
That just creates another problem: if you're taking any measures to reduce tracking (ie. clearing cookies on shutdown or using temporary containers), this won't work. If anubis was being deployed on a site that a user visits often (eg. HN), the user might be convinced to whitelist it, but most anubis deployments are on random blogs or fediverse instances that I might not visit again in months. I'm certainly am not going to whitelist those sites, nor am I going to enable cookies wholesale just to avoid solving challenges.
Or something that shadier websites could calculate ahead of time in your browser, then spend on your behalf.
I feel like Anubis is ironically speedrunning a lot of discoveries the crypto folks have already made several years ago...
I wonder if you could convince the Anubis author to implement x402 payments to bypass the PoW
Then we could implement an exchange, so that if you generate too much Anubis POW, you could exchange it with others.
We could link it to a site, you generate for HN, I for Reddit, but it so happens that you visited Reddit more and I HN, so we depleted our Anubis POW, so we could exchange some Reddit Anubis with some HN Anubis.
And then, we can inextricably link it to a new cryptocurrency -- let's call it BotCoin -- and make Sam Altman pay for it!
Basically reinventing crypto one step at the time.
thanks for trying to make the web suck a little less these days xena!
Anubis is fine if you want to block bots that are bothering you, but I never understood why they don't have robots.txt at the root level too. I happily obey that for my private crawler. I suppose you don't want to duplicate implementations. Fair enough. I guess I should add an Anubis detector so I can just blacklist because that person clearly doesn't want bots.
One of the standard library rules enables robots.txt passthrough by default. I can't control people not choosing to use it.
See also:
https://github.com/eternal-flame-AD/pow-buster - Browser extension that already used WASM to accelerate the anubis solver, among others.
https://github.com/kasper93/anubis_webgpu - Browser extension that uses webgpu, for another order-of-magnitude speedup (depending on your GPU of course).
afaik argon2 should make the GPU less helpful
Argon2 is already GPU-accelerated.
It's wild to me that PoW systems are how we sort the bot problem... the bummer is that all this "work" is just wasted cycles, at least in crypto there is a token you can sell.
Well, there was CoinHive which did this exact thing 6-7 years ago, but that system got abused a bit much
> at least in crypto there is a token you can sell
That would defeat the purpose. The goal is to make scraping costly, not profitable.
Scraping would be costly in this world: scrapers would have to spend tokens in order to get the webpage.
But, in this world, the website owner would receive tokens that they can then use to do whatever they want, including paying for servers and bandwidth. This is the sense in which the cycles aren’t wasted: the website owner now has cash to spend.
Effectively, both scrapers and ordinary users would be paying for the privilege of getting website bytes.
This also solves the problem of having to wait for your phone to solve the challenge while you’re browsing: you can buy or mine some tokens ahead of time and pay them as soon as challenged. So can the scrapers, but because they’re accessing enormous numbers of pages it’s hopefully prohibitively expensive for them.
I guess it could in principle be profitable for the website, not the client?
unfortunately at that point it would be indistinguishable from running cryptojacking on your website
Yup was thinking the same: make honest people pay $0.00001 when they visit the site (in electricity/compute), have the challenge made so that only the website wins a tiny something. Bleed the bots dry.
> This makes Anubis challenges use a memory-hard proof of work function (argon2id) instead of just a CPU hard one. It also means that the "hey Claude vibeslop me a CUDA Anubis solver" route is on its way to being fundamentally dead.
Nice.
Sure, then just replace it with something that is useful to society but not immediately profitable to a scraper, like science research
I want to do this eventually, but it's hard to split things into the micro-tasks that would be required to make this work on Anubis. One of the ideas I'm throwing around is a world where Anubis helps fuzz old games to find timesaves in tool-assisted speedruns. It's harder than you think.
This would be rad.
PoW was originally anti-spam technology.
> hey Claude vibeslop me a CUDA Anubis solver" route is on its way to being fundamentally dead.
Lmao yeah no. I don't think a little argon2 is going to change shit all.
I mean the thesis of Anubis itself is "scrappers are compute limited (in ways that consumer devices are not)" which has its own massive flaws.
The goal isn't to eliminate scrapers, its to prevent a distributed scraping network from requesting 10000 pages a second each from 10000 different websites.
Yes, I also strongly suspect that this is only going to move more parts of scrapers onto consumer devices. The egress proxies are already there, why not use a little bit of the compute as well?