Cloudflare Blocks AI Crawlers by Default for New Domains

Cloudflare, a leading internet infrastructure and website security company, has announced a significant shift in its policy regarding AI crawlers. Effective immediately, Cloudflare will block known AI bots from scraping sites by default for all new domains unless explicit permission is granted or payment is made. This move marks a collective escalation in the industry's response to AI content scraping, with over a million websites already opting into blocking AI crawlers since last fall. The initiative has garnered support from a coalition of major publishers, platforms, and media groups, including the Associated Press, Time, The Atlantic, BuzzFeed, Reddit, Quora, and Universal Music Group.

The decision has been met with widespread approval on social media, with investors and SEO experts praising Cloudflare's initiative. Ed Newton-Rex, former VP of audio at Stability AI, described it as "the way," while pseudonymous trader and content creator Romano RNR noted, "Cloudflare solving problems again." SEO consultant Pedro Dias, who previously worked at Google, commented, "It'll be OK. A force is needed to tilt the balance back to where it once was."

However, the move is not without its critics. Some argue that blocking AI crawlers could potentially harm businesses that rely on AI searches for visibility. Cloudflare CEO Matthew Prince countered this argument by pointing out that a decade ago, Google crawled two pages for every one visit it sent to a publisher. Given the exponential growth in internet usage, publishers should theoretically benefit from increased traffic. However, the reality is that publishers need to make their content easily summarizable by search engines to maintain visibility, which often results in a lopsided ratio of content taken versus traffic received.

Prince highlighted the stark disparity in the ratio of content taken to traffic received for AI companies. For instance, OpenAI's ratio was 250 to 1 six months ago, but it has since worsened to 1,500 to 1. Anthropic, another AI company, had a ratio of 6,000 to 1 six months ago, which has now ballooned to 60,000 to 1. Prince emphasized the need for a new economic model that benefits creators, consumers, and the future of the web. He argued that without proper incentives, content creation will decline, ultimately leading to the demise of the internet as we know it.

To address this issue, Cloudflare is introducing a "pay-per-crawl" marketplace. This innovative approach allows publishers to set their own rates for AI bots, effectively turning the long-dormant HTTP 402 "payment required" code into a digital tollbooth. Publishers can choose to allow free access, charge for each crawl, or block AI crawlers entirely. Initially, this feature is available to leading content creators in early access, with plans to scale the system in the future. This initiative aims to create a sustainable model where AI companies must pay for the content they use, fostering innovation built on permission and partnership.

Cloudflare's move is part of a broader industry trend. Studies by Originality.AI and the Reuters Institute indicate that 48% of the top global websites are already blocking AI crawlers. Cloudflare's new policy ensures that new domains signing up with the company are asked upfront whether they want to allow AI crawlers, with blocking set as the default. Existing customers can change their settings at any time. This collective effort is seen as essential for creating a market with scarcity, which is necessary for a healthy and sustainable digital ecosystem.