On July 3, 2024, Cloudflare - which sits in front of a large fraction of the world’s websites as a content-delivery and security provider - announced a one-click control to block AI scrapers and crawlers. The blog post is titled “Declaring your AIndependence” and states: “we’ve added a brand new one-click to block all AI bots. It’s available for all customers, including those on the free tier.” Site operators enable it by toggling a setting labeled “AI Scrapers and Crawlers” in the Cloudflare dashboard.
What made this different from robots.txt is enforcement. robots.txt is a polite request that a crawler can simply ignore; Cloudflare’s block operates at the network layer and actively refuses identified AI bots, updating automatically as Cloudflare’s systems fingerprint new ones. Cloudflare noted strong demand even against “well-behaved” bots that respect robots.txt, indicating that many site owners wanted a hard stop, not just a posted preference.
This marked an escalation in the standoff over training data. Where GPTBot let individual publishers opt out by editing a file, Cloudflare made blocking effortless and default-available across millions of sites at once - shifting the balance of power toward content owners and making large-scale, unrestricted web scraping meaningfully harder.
Why business readers should care: Cloudflare’s move signaled that the open web’s free buffet of training data is closing. For AI builders, it raises the cost and complexity of data collection; for content owners, it turns “block or license” into a practical, enforceable decision rather than a theoretical one.