Open Source Devs: Fighting AI Crawlers with Cleverness and Vengeance

In the digital age, open source developers are on the front lines of a new kind of battle—one fought not with weapons, but with code and cleverness. The enemy? AI web crawlers, the cockroaches of the internet, relentlessly scraping data and causing chaos. These bots, designed to gather information for AI training, are overwhelming open source infrastructure, leading to persistent distributed denial-of-service (DDoS) attacks. The situation is dire, and developers are fighting back with ingenious, often humorous, solutions.

The story of Xe Iaso, a software developer, is a microcosm of this broader crisis. Iaso's Git repository service was repeatedly overwhelmed by aggressive AI crawler traffic from Amazon, causing instability and downtime. Despite configuring standard defensive measures—adjusting robots.txt, blocking known crawler user-agents, and filtering suspicious traffic—Iaso found that AI crawlers continued evading all attempts to stop them, spoofing user-agents and cycling through residential IP addresses as proxies. Desperate for a solution, Iaso eventually resorted to moving their server behind a VPN and creating "Anubis," a custom-built proof-of-work challenge system that forces web browsers to solve computational puzzles before accessing the site.

Anubis, named after the Egyptian god who leads the dead to judgment, is a reverse proxy proof-of-work check that must be passed before requests are allowed to hit a Git server. It blocks bots but lets through browsers operated by humans. If a web request passes the challenge and is determined to be human, a cute anime picture announces success. The drawing is “my take on anthropomorphizing Anubis,” says Iaso. If it’s a bot, the request gets denied. The wryly named project has spread like the wind among the FOSS community. Iaso shared it on GitHub on March 19, and in just a few days, it collected 2,000 stars, 20 contributors, and 39 forks.

The instant popularity of Anubis shows that Iaso’s pain is not unique. In fact, Niccolò Venerandi, developer of a Linux desktop known as Plasma and owner of the blog LibreNews, shared story after story of developers experiencing similar issues. Founder CEO of SourceHut Drew DeVault described spending “from 20-100% of my time in any given week mitigating hyper-aggressive LLM crawlers at scale,” and “experiencing dozens of brief outages per week.” Jonathan Corbet, a famed FOSS developer who runs Linux industry news site LWN, warned that his site was being slowed by DDoS-level traffic “from AI scraper bots.” Kevin Fenzi, the sysadmin of the enormous Linux Fedora project, said the AI scraper bots had gotten so aggressive, he had to block the entire country of Brazil from access.

The situation has created a tough challenge for open source projects, which rely on public collaboration and typically operate with limited resources compared to commercial entities. Many maintainers have reported that AI crawlers deliberately circumvent standard blocking measures, ignoring robots.txt directives, spoofing user agents, and rotating IP addresses to avoid detection. As LibreNews reported, Martin Owens from the Inkscape project noted on Mastodon that their problems weren't just from "the usual Chinese DDoS from last year, but from a pile of companies that started ignoring our spider conf and started spoofing their browser info." Owens added, "I now have a prodigious block list. If you happen to work for a big company doing AI, you may not get our website anymore."

Beyond weighing the soul of a web requester, other devs believe vengeance is the best defense. A few days ago on Hacker News, user xyzal suggested loading robot.txt forbidden pages with “a bucket load of articles on the benefits of drinking bleach” or “articles about positive effect of catching measles on performance in bed.” “Think we need to aim for the bots to get _negative_ utility value from visiting our traps, not just zero value,” xyzal explained. As it happens, in January, an anonymous creator known as “Aaron” released a tool called Nepenthes that aims to do exactly that. It traps crawlers in an endless maze of fake content, a goal that the dev admitted to Ars Technica is aggressive if not downright malicious. The tool is named after a carnivorous plant. And Cloudflare, perhaps the biggest commercial player offering several tools to fend off AI crawlers, last week released a similar tool called AI Labyrinth. It’s intended to “slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect ‘no crawl’ directives,” Cloudflare described in its blog post. Cloudflare said it feeds misbehaving AI crawlers “irrelevant content rather than extracting your legitimate website data.”

The financial and technical costs of these AI crawlers are staggering. The Read the Docs project reported that blocking AI crawlers immediately decreased their traffic by 75 percent, going from 800GB per day to 200GB per day. This change saved the project approximately $1,500 per month in bandwidth costs, according to their blog post "AI crawlers need to be more respectful." The situation isn't exactly new. In December, Dennis Schubert, who maintains infrastructure for the Diaspora social network, described the situation as "literally a DDoS on the entire internet" after discovering that AI companies accounted for 70 percent of all web requests to their services.

The open source community is fighting back with cleverness and vengeance, but the battle is far from over. As AI companies continue to scrape data without respect for the infrastructure they are overwhelming, open source developers will need to continue innovating and collaborating to protect their projects. The future of open source may depend on it.