The Open Web Spent Ten Days Building Walls and Could Not Agree Who Was Outside

Between 6 and 16 September, four separate stories about keeping robots out of websites hit the Hacker News front page. Anubis, the proof-of-work bot wall, shipped WebAssembly (362 points, 199 comments). Google replaced every search result link with an opaque redirect (678 points, 553 comments). The Internet Archive explained why the Wayback Machine has been throwing 429 errors (590 points, 319 comments). And Cloudflare shipped a setting that lets a site refuse AI training while staying in search results (79 points, 44 comments).

That is 1,709 points and 1,115 comments in ten days on what is functionally one question. We read all four threads looking for a consensus, because a community that spends a thousand comments on the same subject usually lands somewhere. This one did not. It produced four different answers about who the enemy even is.

The argument, at its strongest on both sides

The case for the walls is not complicated and it is not stupid. Free infrastructure is being consumed at industrial scale by companies that will never pay for it, the load is real, and the people running the servers are volunteers and non-profits. The Internet Archive's own framing was that it has "been hit by waves of high-volume automated traffic" and put protections in place to keep the service running. If the choice is between a 429 for some humans and the Wayback Machine falling over, that is not a close call.

The case against is also not stupid, and it is the one the threads kept circling. Every mechanism in this list works by making the web harder to read programmatically, which means harder to read at all for anyone without a mainstream browser, a residential IP, JavaScript, and now WebAssembly. The people who get locked out first are not the scraping farms. They are the ones on a VPN, behind CGNAT, on airport WiFi, or running an old browser.

The takes

simonw supplied the theory that most of the Wayback thread ran with:

I'm pretty certain this is scrapers that are trying to workaround blocks on accessing original sites by hitting the Wayback Machine copy instead. Appalling behavior. In addition to the load it puts on this vital non-profit piece of Internet infrastructure, we've also already seen some sites opt out of the Wayback Machine to prevent their content from being scraped via this alternative route.

That last clause is the quiet horror of the whole story: the archive is losing content because it is being used as a scraping bypass. But the thread did not simply accept the premise. pelican0 asked for evidence:

Is it established that the scraping scourge of late is primarily driven by AI companies? Anyone aware of any relevant studies?

Nobody in the replies produced one. Meanwhile timpera described the cost being paid by people who are definitely not scrapers:

from my residential IP, simply moving the mouse on the calendar for a specific URL is enough to get stuck on 429 error messages for a while; and from corporate ISPs (for example, on airport WiFi), you often can't access the WM at all.

Two commenters went straight to paywalls. ilamont asked "Shouldn't the solution be to gate bulk access for automated services for a price?" and petterroea volunteered: "I'd be happy to pay a 5$/month donation to get a higher rate limit/more lenient filter put on me." Which is a reasonable offer and also, if you squint, a subscription to the public record.

Over in the Google thread, mrkramer landed the line the whole week was building toward:

They are allowed to scrape everybody else but get their feelings hurt when they get scraped....oh yea they respect robots.txt. Guess what, scraping everything that is public is legal.

In the Anubis thread, chews questioned the entire defensive mechanism rather than its target:

It's wild to me that PoW systems are how we sort the bot problem... the bummer is that all this "work" is just wasted cycles, at least in crypto there is a token you can sell.

And in the same thread, doctor_radium pointed out that he runs Firefox with WebAssembly disabled on purpose and asked, in his words "in a small voice", for a message explaining why the challenge failed. The bot wall's newest feature locks out a careful user who made a deliberate privacy choice.

The Cloudflare thread was the most cynical of the four. nirmeetimthebes:

"Accountable" is just a fancy word for "pinky promise, but with a label." Nothing stops the data from ending up in a training run once it's already been fetched.

And qsbuilder found the structural flaw in the product itself:

The irony is that search engines are AI companies now. Telling them 'index me for search but don't train your models' is asking them to split a brain that's already fully merged.

The thing only cross-reading finds

userbinator posted in two of these threads, nine days apart, making the same argument both times, and in neither thread did anyone connect them.

On the Google redirect: "Since the article brings up 'AI scrapers', I'll just point it out as being the latest scare-tactic for coercing people to give up the privacy, anonymity, and (browser) freedom of an open interoperable Internet." On the Wayback Machine: "Thank you for not immediately blaming it on 'AI bots'. I suspect there's some entity manufacturing consent for strong identity/age verification/sanctioned-browser-OS 'walled garden' Internet."

We are not endorsing the manufactured-consent reading; it is unfalsifiable as stated, and the Internet Archive's load problem is obviously real. But the narrower observation underneath it survives the skepticism: "AI scrapers" has become a justification that no longer gets asked for evidence. pelican0 asked and got nothing back. That is worth noticing regardless of what you think the answer is.

The four threads also converged on four different villains, which is the actual finding here. The Anubis thread blamed the economics of proof-of-work. The Google thread blamed Google. The Wayback thread blamed AI companies, then argued about whether it should. The Cloudflare thread blamed Cloudflare for selling the cure and the disease, in gchamonlive's phrasing: "Cloudflare, enabling the problem and the solution since, how long has it been?" Same wall, four enemies.

The best comment nobody replied to

The Internet Archive asked blocked users to email info@archive.org with their operating system, browser and IP address. msephton, one reply, no follow-up:

Why can't they capture OS, Browser, and IP address at the time of error? All that information is available at the point of failure, the user should not need to email it in.

Three hundred and nineteen comments about the philosophy of the open web, and the one person who noticed that the 429 page already knows everything the support email is asking for got almost no engagement. That is a small, shippable fix sitting under an argument about enclosure.

Honourable mention to JohnTHaller in the Google thread, who checked who wrote the anti-scraping article everyone was upvoting and noted that the publisher appears to scrape Google results and sell them through an API. He hedged it carefully as his own quick read. Nobody built on it.

Our read

The pro-wall side is right about the immediate problem and wrong about where it ends. The anti-wall side is right about where it ends and has no answer for next Tuesday.

Every one of these defences is a filter on capability, not on intent, and capability is exactly what the scrapers have more of than you do.

Proof-of-work costs an attacker with a GPU fleet nothing and costs a user on an old phone real seconds. Rate limiting by IP hits CGNAT users and airport WiFi long before it hits a distributed crawler. Cloudflare's "Accountable" label is enforced by corporate promise, and the same post notes that 17 percent of sites already enable some mechanism to block training while under 1 percent block search, which tells you people want this distinction badly enough to accept a pinky promise.

What would change our mind: hard numbers. If the Internet Archive published request volumes by ASN and user agent before and after, the argument would be settled in an afternoon. Nobody has that data publicly, which is why a thousand smart people spent ten days reasoning from vibes, and why we are writing about the disagreement rather than resolving it.

One honest limit on this piece. Reddit's search returned 403 for every query we tried, and the RSS fallback gave us marketing spam rather than r/programming, so this is a Hacker News read with no Reddit cross-check. If those subreddits landed somewhere different, we did not see it.

Which side are you funding

The concrete question, because it splits people who agree on everything else: would you pay the 5 dollars a month petterroea offered for a higher Wayback Machine rate limit? If yes, you have accepted that the public record is a subscription. If no, say what funds it instead. We genuinely do not have a good answer and are not going to pretend otherwise.

If you want the specific story that made 553 people argue about this, we covered Google replacing every search result link with an opaque redirect when it happened. It is the single clearest example of a wall that was announced as anti-scraping and lands hardest on people just trying to copy a link.