Look, I want to see WCAG adopt "Any ad that covers content, a cookie banner, or Anubis or Cloudflare harassment is an immediate F grade" I'm a little more hopeful of that than to get a requirement like "you should actually have a disabled person test it"
What the people who like Anubis don't get is that the cost for honest people is very real, the cost for crawlers is basically $0 because: most of the people who run crawlers are just idiots and aren't paying attention to what's happening and get crap data and don't even know they are getting bounced and probably don't even look at the crap data they collect, the worst of the crawlers are running botnets using stolen CPU, the real AI companies don't have any financial discipline and wouldn't care it if anubis wasted a CPU*hour per page.
Or rather, those "click on the bus" images raise your blood pressure but are close to trivial for a bot to automate... It's all performative... We're good open source people who prove we are good open source people because we put Anubis on our page to shame those bad AI people.
Interesting, if I understand correctly it seems like Anubis is just being used in a use case inappropriate way here?
The Arch equivalent to this would be building AUR packages from source, which seems to universally work fine without any of these issues.
Are NixOS repos putting their source behind Anubis or something? I’m tempted to start on an “I don’t like NixOS” rant but I don’t know who owns these forges.
then some process takes those PKGBUILDs, downloads the sources from the upstream URLs, and builds them into the binary packages that end up in the official pacman repositories.
The same process happens on NixOS to build their official repository (which is called the "Nix Binary Cache"). So most NixOS users, when installing software, they are just downloading prebuilt binary packages from official NixOS servers just like pacman does on Arch. I, however, disable the Nix's official package repository so when I install software it is like building from a PKGBUILD: it needs to download the sources from the upstream URL and build the package.
I'd imagine the reason I run into this issue a lot more than you do is usually you only install a couple of packages from the AUR. The vast majority of your packages are going to come prebuilt from official Arch repositories. Whereas on my systems, 100% of the software is being built from source. So if any of my software is developed on a site using anubis to protect their source tarballs or patch files, I will hit this issue.
For me to open 5 pages, which is more than often when I search with google etc, that all have Anubis, will take more time for them to load than that I ask chatgpt and it generates an answer from web search, at least in some cases. It is 100% virtue signaling that has no technical fundamentals to deter scrappers, which likely implement better solvers and have virtually unlimited amount of time and compute to spare, it just punishes me who have a body and a limited time budget. Humiliating even for a split of a second. Now I just skip any website that I do not care enough and has Anubis; the ones I have to visit, I will remember their mindless choice forever.
We can make up unrealistic scenarios all day long but I just don’t believe for a second that anyone is this inconvenienced by it.
The idea that you’re in desperate need to get information from 5 web pages as quickly as possible like you’re a contestant on The Amazing Race is ridiculous. This is the definition of dramatic.
Anubis finishes up in less time than takes to read this sentence.
Do it 20 times in the search for one article, and encounter at least a 30% misconfiguration rate (inappropriately high challenge value), and it does begin to bear out. If you are looking for obscure or poorly managed data, it unfortunately does become an issue.
In the daily, if you spend time researching with plenty of dead ends, ot does bear out in lost time. Not by dollars, but nickles and dimes.
I did not use chatbot for search until Anubis, Cloudflare and recaptcha are so prevalent that chatbot literally parses through the pages faster than I can open any one of them, using my daily browser, with my own hands. It seems not only my choice does not matter, but most websites also do not care about human readers. At this point I just use chatbot as search engine out of vengeance.
Also, optimized Anubis & other proof of work challenge solvers are much faster than a stock browser.
Can be faked. Record people doing it hundreds if not thousands of times and make a statistical model. Google and Facebook of course would love it if your were logged in and could look at the muddy footprints you leave all over the web. Yet another way to punish people for using the wrong browser or not logging in.
I've worked at multiple AI browser companies at this point and I think the best way forward is to start paying for content again honestly. Brave and BAT were too early but we need more things like that with lower friction and more primitives baked into the browser. The AI bots that create value for people and have money will happily pay to get access to the content, keeping it alive in the process and paying for the maintenance cost of allowing good bots.
DBSC, DPoP, and Google's latest "Google Cloud Fraud Defense" reCAPTCHA using phone are another approach where you use hardware secure enclaves to prevent cookies from leaving devices and attest to 1 visitor = 1 physical device. I don't like these because I want my personal agents to be able to access the things I can on my behalf. If I misbehave then block me & bots, not all bots, this is not as impossible as companies make it sound.
The worst possible approach is to require biometric or government ID, that's the one we all need to push hard against, as it's bad for everyone involved.
> paying for content [...] The worst possible approach is to require biometric or government ID,
IMO a good middle-ground is a "proof of stake", where you just need to prove you're an entity that has some meaningful incentive to follow the rules and be nice, because you'll take some damage if the site tries to terminate the relationship.
That differs from other options like paying the site money (perverse incentive problem) or proving a relationship to a unique human (privacy dangers) or that you're a human and not a robot (difficult to test.)
Imagine a vending-machine at city-hall, where anyone can spend $5 to a legally-recognized charity of their choice. In exchange, they get some anonymous tokens [0] can be loaded into a web browser for use.
The point is we can all benefit by strategically relaxing requirements.
[0] Technical details of the anonymity and deterring resale etc. are important, but I'm gonna shelve them for the moment in the interests of space/brevity.
Most of the time a web page reports an error or seems to be down, or it gives you another captcha no matter how many times you correctly answer it, it's just denying you access, but it's also trying to hide that from you. Whether there's captchas or an outright denial, those pages are guarded by bots that only allow other bots to proceed to the page. We've known for several years that captchas only deter humans (https://www.newscientist.com/article/2384228-bots-are-better...) so clearly anyone currently using them doesn't want humans visiting the page.
Most of the time, the Gemini summary that Google includes in their search results isn't particularly useful, but extracting data from those bot-only pages is one thing it excels at. So, whenever a search result links to a page guarded by a bot, and it senses that I am a human and denies access, I go back to the Google search results page, and ask Gemini to scrape the page and provide the data I want from it.
Do you use a VPN or Tor, or do something that might make you look like bot traffic, like always using Incognito mode, spoofing your User Agent, etc? Are you connecting from something other than a residential IP?
You're having to prove you're human all the time because there's likely something you're doing that makes you look like a bot.
Living in a Non-US, Non-Canadian or Non-European country will basically make most CDNs and similar services flag you immediately as a "probably bot traffic" and you'll get served endless captchas and similar hassles.
It's even worse when you are traveling cause then you run into country-blocks on websites and videos on top of all of that, and you didn't even mention the forced logging-in or registration walls to continue browsing/shopping/etc!
Feels like the whole internet is hitting that hockey stick growth... towards becoming shit.
I've been starting to feel this as well. I isolate tracking cookies from the big ad networks and it shows.
Another one on the list now is Amazon adding amazon.com redirects around IMDB pages. I can't even follow IMDB links anymore without getting trapped in broken redirects, because my Amazon cookies are in another session and whatever challenge they are running on amazon.com is bad at completing redirects and keeps redirecting brokenly encoded addresses that are 404 pages instead. I have to search things from IMDB in an Amazon container to get it to take me to the same addresses the direct link should have and used to in any container.
This has been particularly annoying for me because I disable NixOS's binary cache and build all my software from source. So when I run `nix build` I end up fetching source code tarballs from code forges. Invariably, the build will fail because what was supposed to be thesourcecode.tar ended up being the html for the anime girl page, leading to a hash mismatch.
I haven't yet figured out the incantation to download the source files myself and inject them into the nix store. `nix-store --add-fixed sha256 <file>` always seems to result in a different nix store path than the one `nix build` was downloading to.
I totally understand the desire to stem the fire hose of AI crawlers. I self-host my code forge and so I am no stranger to the devastation those locusts leave in their wake. But this is also hurting innocent non-crawling humans who are just trying to compile your software.
Yes, I feel like there has been a sharp uptick in this the past couple months.
Whatever magic is built into the CloudFlare checkbox and my motion to click on it, I really wish was just part of the browser and my motion to click on everything else.
I've also always had the feeling my browser setup is what's causing it. I don't think Brave is a "weird" browser, but its privacy oriented features is what Cloudflare challenges and CAPTCHA unfortunately don't play well with.
That was happening on our websites at work, and users started complaining about it. The problem was due to the cloudflare default settings, which we dialed down in response to the complaints.
I think for web sites you really love and care about, you should write to them and let them know that the user experience sucks. If enough people complain they may fix the captchas like we did!
Once you get past that, it's on to the auth walls where you have to login every day or week. We have perfect the nagging modals to suit all anti-consumer patterns.
This is the internet in 2026 and it sure as hell won't get any better.
Kinda disappointing that this doesn't get daily attention on HN of all places.
I think most people don't realize that the internet is effectively partitioned already, on multiple levels, some intentional, some unintentional. Language being the biggest one, anglosaxon internet is just one of many, probably the one we're the most familiar with, but there are 4-5 others out there, that are as big and bigger than the one we're all used to.
Secondly, depending on what continent and country you use the internet from, you get a vastly different experience. If you're unlucky enough to live in Brazil, Peru, Algeria, Cambodia or hundreds of other countries, you'll be served challenge after challenge, regardless of how many times you continue to pass them. Cloudflare will in some cases just straight up reject traffic from you, and many times just the fact that websites set too short timesouts to HTTP requests, means you effectively can't browse their websites, even if you'd be ready to wait 2 seconds for each request to be done.
So you end up segregating internet users per country, because of gatekeepers like Cloudflare and others, who've somehow made people feel much more comfortable with just blocking all traffic from a whole country because of some attack they sustained in the past originating from that country.
Not sure what the solution is, but I've definitely felt the effects of these things too, making the internet I'm used to borderline impossible to use because of the country I was in. I'm not sure most people realize how fragmented it all already is, and been for decades already, not least because of language.
If I dont get inside the page in 5-10 seconds tops I hit the back button and continue to read the HN comments ( about the article). I rather not give them traffic if they treat my network request like I am e-begging to read the article.
I have two Firefox extensions that solve captchas for me. They are just eye exams and most unwelcome.
Buster: Captcha Solver for Humans [0]
NopeCHA: CAPTCHA Solver [1]
OF COURSE, these are band-aids on an internet under full assault.
P.S. Paywall busters are a nuisance to run through. Often, a paywalled article is syndicated (MSN does this a lot) or available at arxiv, or an author's institution, so "Use the web search, Luke."
The original model for web content distribution wasn't designed with Cloudflare style reverse proxies and they weren't a thing until much later. The original model is for the most part dead, sans some quiet corners here and there.
The original model for web didn't include REST style API calls.
A new model designed holistically for content distribution, with a hardened standardized spec for the API calls part, something that by design entirely shirks today's daily challenges, is necessary.
_
> Or just come back read the comments on hn for that article.
Look, I want to see WCAG adopt "Any ad that covers content, a cookie banner, or Anubis or Cloudflare harassment is an immediate F grade" I'm a little more hopeful of that than to get a requirement like "you should actually have a disabled person test it"
What the people who like Anubis don't get is that the cost for honest people is very real, the cost for crawlers is basically $0 because: most of the people who run crawlers are just idiots and aren't paying attention to what's happening and get crap data and don't even know they are getting bounced and probably don't even look at the crap data they collect, the worst of the crawlers are running botnets using stolen CPU, the real AI companies don't have any financial discipline and wouldn't care it if anubis wasted a CPU*hour per page.
Or rather, those "click on the bus" images raise your blood pressure but are close to trivial for a bot to automate... It's all performative... We're good open source people who prove we are good open source people because we put Anubis on our page to shame those bad AI people.
What is the cost of Anubis for honest people? I’m legitimately confused by that one.
I don’t like the cloudflare checkbox or click all the buses to access this website but Anubis you just sit there for a split second and do nothing.
That’s what people like about Anubis.
> What is the cost of Anubis for honest people?
I just made a post of one of the costs of Anubis for honest people before I saw your comment. Linking it here instead of copying: https://news.ycombinator.com/item?id=50358838
Interesting, if I understand correctly it seems like Anubis is just being used in a use case inappropriate way here?
The Arch equivalent to this would be building AUR packages from source, which seems to universally work fine without any of these issues.
Are NixOS repos putting their source behind Anubis or something? I’m tempted to start on an “I don’t like NixOS” rant but I don’t know who owns these forges.
It's sort of a mix between AUR and regular pacman repositories. On Arch, you have PKGBUILDs that list upstream URLs for your package sources. For example: https://gitlab.archlinux.org/archlinux/packaging/packages/ap...
then some process takes those PKGBUILDs, downloads the sources from the upstream URLs, and builds them into the binary packages that end up in the official pacman repositories.
The same process happens on NixOS to build their official repository (which is called the "Nix Binary Cache"). So most NixOS users, when installing software, they are just downloading prebuilt binary packages from official NixOS servers just like pacman does on Arch. I, however, disable the Nix's official package repository so when I install software it is like building from a PKGBUILD: it needs to download the sources from the upstream URL and build the package.
I'd imagine the reason I run into this issue a lot more than you do is usually you only install a couple of packages from the AUR. The vast majority of your packages are going to come prebuilt from official Arch repositories. Whereas on my systems, 100% of the software is being built from source. So if any of my software is developed on a site using anubis to protect their source tarballs or patch files, I will hit this issue.
For me to open 5 pages, which is more than often when I search with google etc, that all have Anubis, will take more time for them to load than that I ask chatgpt and it generates an answer from web search, at least in some cases. It is 100% virtue signaling that has no technical fundamentals to deter scrappers, which likely implement better solvers and have virtually unlimited amount of time and compute to spare, it just punishes me who have a body and a limited time budget. Humiliating even for a split of a second. Now I just skip any website that I do not care enough and has Anubis; the ones I have to visit, I will remember their mindless choice forever.
We can make up unrealistic scenarios all day long but I just don’t believe for a second that anyone is this inconvenienced by it.
The idea that you’re in desperate need to get information from 5 web pages as quickly as possible like you’re a contestant on The Amazing Race is ridiculous. This is the definition of dramatic.
Anubis finishes up in less time than takes to read this sentence.
Do it 20 times in the search for one article, and encounter at least a 30% misconfiguration rate (inappropriately high challenge value), and it does begin to bear out. If you are looking for obscure or poorly managed data, it unfortunately does become an issue.
In the daily, if you spend time researching with plenty of dead ends, ot does bear out in lost time. Not by dollars, but nickles and dimes.
I did not use chatbot for search until Anubis, Cloudflare and recaptcha are so prevalent that chatbot literally parses through the pages faster than I can open any one of them, using my daily browser, with my own hands. It seems not only my choice does not matter, but most websites also do not care about human readers. At this point I just use chatbot as search engine out of vengeance.
Also, optimized Anubis & other proof of work challenge solvers are much faster than a stock browser.
It doesn't matter if you have to wait 10 minutes for Anubis to load, it will eventually load.
If you get blocked by another anti-bot mechanism, then there's nothing you can do without compromising on privacy or browser/device choice somehow.
An open web is more important than a convenient web, and Anubis or similar tools are the only option that keeps things truly open.
I think thought user behaviour telemetry was a big part of the captcha recipe. Not so much the clicking of the right squares per se
Can be faked. Record people doing it hundreds if not thousands of times and make a statistical model. Google and Facebook of course would love it if your were logged in and could look at the muddy footprints you leave all over the web. Yet another way to punish people for using the wrong browser or not logging in.
speaking of WCAG https://news.coffee/wcag.gif
We can call the captchas and lockouts harassment, but I don't think a 3 second delay is in the same category of problem.
It's going to get worse before it gets better:
- https://docs.sweeting.me/s/ai-scraping-will-force-us-towards...
- https://docs.sweeting.me/s/so-you-want-to-build-a-social-net...
I've worked at multiple AI browser companies at this point and I think the best way forward is to start paying for content again honestly. Brave and BAT were too early but we need more things like that with lower friction and more primitives baked into the browser. The AI bots that create value for people and have money will happily pay to get access to the content, keeping it alive in the process and paying for the maintenance cost of allowing good bots.
DBSC, DPoP, and Google's latest "Google Cloud Fraud Defense" reCAPTCHA using phone are another approach where you use hardware secure enclaves to prevent cookies from leaving devices and attest to 1 visitor = 1 physical device. I don't like these because I want my personal agents to be able to access the things I can on my behalf. If I misbehave then block me & bots, not all bots, this is not as impossible as companies make it sound.
The worst possible approach is to require biometric or government ID, that's the one we all need to push hard against, as it's bad for everyone involved.
> paying for content [...] The worst possible approach is to require biometric or government ID,
IMO a good middle-ground is a "proof of stake", where you just need to prove you're an entity that has some meaningful incentive to follow the rules and be nice, because you'll take some damage if the site tries to terminate the relationship.
That differs from other options like paying the site money (perverse incentive problem) or proving a relationship to a unique human (privacy dangers) or that you're a human and not a robot (difficult to test.)
Imagine a vending-machine at city-hall, where anyone can spend $5 to a legally-recognized charity of their choice. In exchange, they get some anonymous tokens [0] can be loaded into a web browser for use.
The point is we can all benefit by strategically relaxing requirements.
[0] Technical details of the anonymity and deterring resale etc. are important, but I'm gonna shelve them for the moment in the interests of space/brevity.
I call those bot-only web pages.
Most of the time a web page reports an error or seems to be down, or it gives you another captcha no matter how many times you correctly answer it, it's just denying you access, but it's also trying to hide that from you. Whether there's captchas or an outright denial, those pages are guarded by bots that only allow other bots to proceed to the page. We've known for several years that captchas only deter humans (https://www.newscientist.com/article/2384228-bots-are-better...) so clearly anyone currently using them doesn't want humans visiting the page.
Most of the time, the Gemini summary that Google includes in their search results isn't particularly useful, but extracting data from those bot-only pages is one thing it excels at. So, whenever a search result links to a page guarded by a bot, and it senses that I am a human and denies access, I go back to the Google search results page, and ask Gemini to scrape the page and provide the data I want from it.
Do you use a VPN or Tor, or do something that might make you look like bot traffic, like always using Incognito mode, spoofing your User Agent, etc? Are you connecting from something other than a residential IP?
You're having to prove you're human all the time because there's likely something you're doing that makes you look like a bot.
bad device on the network running residential proxy malware would be my guess
GreyNoise offers a check for this, at least if it’s something that is scanning the internet: https://check.labs.greynoise.io/
Living in a Non-US, Non-Canadian or Non-European country will basically make most CDNs and similar services flag you immediately as a "probably bot traffic" and you'll get served endless captchas and similar hassles.
It's even worse when you are traveling cause then you run into country-blocks on websites and videos on top of all of that, and you didn't even mention the forced logging-in or registration walls to continue browsing/shopping/etc!
Feels like the whole internet is hitting that hockey stick growth... towards becoming shit.
I've been starting to feel this as well. I isolate tracking cookies from the big ad networks and it shows.
Another one on the list now is Amazon adding amazon.com redirects around IMDB pages. I can't even follow IMDB links anymore without getting trapped in broken redirects, because my Amazon cookies are in another session and whatever challenge they are running on amazon.com is bad at completing redirects and keeps redirecting brokenly encoded addresses that are 404 pages instead. I have to search things from IMDB in an Amazon container to get it to take me to the same addresses the direct link should have and used to in any container.
> weird anime girl
This has been particularly annoying for me because I disable NixOS's binary cache and build all my software from source. So when I run `nix build` I end up fetching source code tarballs from code forges. Invariably, the build will fail because what was supposed to be thesourcecode.tar ended up being the html for the anime girl page, leading to a hash mismatch.
I haven't yet figured out the incantation to download the source files myself and inject them into the nix store. `nix-store --add-fixed sha256 <file>` always seems to result in a different nix store path than the one `nix build` was downloading to.
I totally understand the desire to stem the fire hose of AI crawlers. I self-host my code forge and so I am no stranger to the devastation those locusts leave in their wake. But this is also hurting innocent non-crawling humans who are just trying to compile your software.
I've noticed a lot more Cloudflare human checks recently. I kind of assumed it was sites trying to avoid being scraped to death for training data.
Alternatively, I use Firefox with uBlock Origin and Privacy Badger - so it might be that.
Yes, I feel like there has been a sharp uptick in this the past couple months.
Whatever magic is built into the CloudFlare checkbox and my motion to click on it, I really wish was just part of the browser and my motion to click on everything else.
I'll ask it but what about your browsing setup is unusual?
Because often these reports are from people using weird browsers, privacy plugins, VPNs, javascript off, etc
It does happen to "normal" browsers but are you doing anything that look suspicious?
> these reports are from people using weird browsers, privacy plugins, VPNs, javascript off, etc
Best description of the Cloudflare groupthink.
Not using Chromium based browser, not paying Apple or Microsoft, actually enforcing do not track me policies with add-ons. Bunch of weirdos indeed.
I've also always had the feeling my browser setup is what's causing it. I don't think Brave is a "weird" browser, but its privacy oriented features is what Cloudflare challenges and CAPTCHA unfortunately don't play well with.
That was happening on our websites at work, and users started complaining about it. The problem was due to the cloudflare default settings, which we dialed down in response to the complaints.
I think for web sites you really love and care about, you should write to them and let them know that the user experience sucks. If enough people complain they may fix the captchas like we did!
[flagged]
Once you get past that, it's on to the auth walls where you have to login every day or week. We have perfect the nagging modals to suit all anti-consumer patterns.
This is the internet in 2026 and it sure as hell won't get any better.
Kinda disappointing that this doesn't get daily attention on HN of all places.
I get the fatigue too but feel if cloudflare is effective then the benefits outweigh the concerns. Bots will spam everything so hard
It really makes me feel like I'm back in the early 00's with dial-up again (derogatory)
Friction, friction, more friction = FIRE!
IMHO -- I am tired of it too. I hate it.
I think most people don't realize that the internet is effectively partitioned already, on multiple levels, some intentional, some unintentional. Language being the biggest one, anglosaxon internet is just one of many, probably the one we're the most familiar with, but there are 4-5 others out there, that are as big and bigger than the one we're all used to.
Secondly, depending on what continent and country you use the internet from, you get a vastly different experience. If you're unlucky enough to live in Brazil, Peru, Algeria, Cambodia or hundreds of other countries, you'll be served challenge after challenge, regardless of how many times you continue to pass them. Cloudflare will in some cases just straight up reject traffic from you, and many times just the fact that websites set too short timesouts to HTTP requests, means you effectively can't browse their websites, even if you'd be ready to wait 2 seconds for each request to be done.
So you end up segregating internet users per country, because of gatekeepers like Cloudflare and others, who've somehow made people feel much more comfortable with just blocking all traffic from a whole country because of some attack they sustained in the past originating from that country.
Not sure what the solution is, but I've definitely felt the effects of these things too, making the internet I'm used to borderline impossible to use because of the country I was in. I'm not sure most people realize how fragmented it all already is, and been for decades already, not least because of language.
How long do you spend on the page?
If I dont get inside the page in 5-10 seconds tops I hit the back button and continue to read the HN comments ( about the article). I rather not give them traffic if they treat my network request like I am e-begging to read the article.
That sounds about right. Maybe they track that. The data gets fed up to Cloudflare general and diffuses back out to the operating sites.
I have two Firefox extensions that solve captchas for me. They are just eye exams and most unwelcome.
Buster: Captcha Solver for Humans [0]
NopeCHA: CAPTCHA Solver [1]
OF COURSE, these are band-aids on an internet under full assault.
P.S. Paywall busters are a nuisance to run through. Often, a paywalled article is syndicated (MSN does this a lot) or available at arxiv, or an author's institution, so "Use the web search, Luke."
[0] https://github.com/dessant/buster
[1] https://nopecha.com/
Completely agreed!
The original model for web content distribution wasn't designed with Cloudflare style reverse proxies and they weren't a thing until much later. The original model is for the most part dead, sans some quiet corners here and there.
The original model for web didn't include REST style API calls.
A new model designed holistically for content distribution, with a hardened standardized spec for the API calls part, something that by design entirely shirks today's daily challenges, is necessary.
_
> Or just come back read the comments on hn for that article.
Same lol
i get that if i have a vpn on. so i keep my vpn off. which is bad. oopsie!
[dead]