Cloudflare Blocks AI Crawlers By Default To Empower Publishers

Cloudflare, a prominent internet infrastructure company responsible for routing approximately 20% of global web traffic, has announced a significant policy change. Effective from Tuesday, the company will begin blocking artificial intelligence (AI) crawlers by default. This move comes in response to publishers' demands for greater control and compensation for their data, which has been extensively scraped by AI companies for training and other purposes.

The content delivery network (CDN) plays a crucial role in caching and serving data closer to users, thereby enhancing website performance. With the new policy, any new domain signing up for Cloudflare services will be prompted to decide whether AI bots can access their content and under what conditions. This decision can be made at the time of signup, allowing site owners to either permit controlled access or block scrapers entirely.

This change builds on Cloudflare’s earlier initiatives aimed at empowering publishers with more control over their data. Last year, the company introduced a one-click solution to block all known AI bots and a dashboard to monitor crawler activity. These tools enable site owners to distinguish between different types of crawlers, such as those scraping data for AI training, search purposes, or other uses. The latest announcement formalizes these protections and enforces them by default, giving publishers more autonomy over their content.

Cloudflare CEO Matthew Prince emphasized the company's goal to "put the power back in the hands of creators, while still helping AI companies innovate." This initiative is underpinned by Cloudflare’s Pay per Crawl system, a marketplace where AI companies and content owners can agree on compensation per access. Both parties must have Cloudflare accounts, and once set up, they can negotiate prices and terms for web crawling activities. Cloudflare acts as a broker in the transaction, charging the AI company and passing the earnings to the publisher.

Several AI developers, including OpenAI, the Microsoft-backed artificial intelligence firm behind ChatGPT, have declined to participate in the program. OpenAI criticized Cloudflare for inserting a new intermediary between publishers and AI developers, asserting that it has a history of honoring the robots.txt protocol, which allows website operators to control crawler access. OpenAI also insisted that it respects site preferences and has been accused of overwhelming websites and significantly impacting user experience.

In a June analysis, Cloudflare claims to have found a gap between scraping frequency and traffic referrals. For example, Google’s crawler accessed websites 14 times for every visit it sent back, while OpenAI’s bot scraped sites 17,000 times for every referral. This discrepancy highlights the potential intrusiveness of AI crawlers and their impact on user experience. If Cloudflare’s system works as intended, it could limit the ability of AI chatbots to collect and train on large-scale web data.

Major media companies have rallied behind Cloudflare’s efforts to reclaim control over digital content. Publishers, including TIME, The Associated Press, Conde Nast, The Atlantic, ADWEEK, and Fortune, have all agreed to block AI bots by default. These media outlets have traditionally accepted data scraping from platforms like Google in exchange for traffic and ad revenue. However, the current AI-driven ecosystem lacks such reciprocity, with AI platforms like ChatGPT and Claude consuming content without meaningful engagement or revenue for original sources.

Cloudflare has stated that it will continue to work with developers to push AI crawlers that wish to be allowed access to disclose their identity, purpose, and crawling behavior. CEO Matthew Prince reiterated the importance of protecting original content, stating, “Original content is what makes the Internet one of the greatest inventions in the last century. We have to come together to protect it.”