Internet infrastructure firm Cloudflare will now default to blocking AI scraper bots from accessing original content on client websites without permission or compensation.

AI has changed the internet. Before, when a user typed a query into a search engine, they were presented with a series of links. Clicking on one would direct them to a website, and this exchange generated traffic and ad revenue for content creators.

Nowadays, AI chatbots such as ChatGPT are being used instead of traditional search. These chatbots are trained on data scraped by AI bots that vacuum up information from the internet.

AI scraper bots or crawlers often take this data without permission, with content creators seeing almost no traffic and therefore almost no value.

Cloudfare, which powers more than 20% of the internet (15 million domains), is now putting a stop to this by changing the default to block AI crawlers unless they pay creators for their content. 

Cloudfare’s new tech means its clients can now choose if they want AI crawlers to access content on their websites, and decide how AI companies can use that content. 

In a new permission-based model, AI companies must also state what their crawlers are using the data for – training, inference or search – to help website owners decide whether they will allow or deny scraping of their content.

Matthew Prince, co-founder and CEO of Cloudflare, said: “AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. 

“This is about safeguarding the future of a free and vibrant internet with a new model that works for everyone.”

With this new technology, each new domain will default to blocking crawlers. This eliminates the need for webpage owners to manually configure their settings to opt out.

They can always then choose to enable crawling at any time from the settings if they want their content to be freely accessed.

This move to “value original content” has been very much welcomed by Cloudflare’s vast client base.

Roger Lynch, CEO of Condé Nast, said: “This is a critical step towards creating a fair value exchange on the internet that protects creators, supports quality journalism and holds AI companies accountable.”

Prince said: “Cloudflare’s mission is to help build a better internet. I’m proud of the role we’re playing in doing exactly that as the web evolves. And I’m proud that we’re helping content creators stick up and demand value for the content they worked hard to create.”

In April 2025, Wikimedia Foundation revealed that AI scraper bots are putting costly strain on Wikimedia infrastructure.