Cloudflare blocks ChatGPT and other AI bots from ‘ripping off’ content, ‘saving’ online publishers
Cloudflare, the internet’s silent giant, has barred AI companies pillaging online content, declaring it isn’t just about protecting publishers but safeguarding the very best of the internet.
A major internet player has declared war on dubious AI bots.
Cloudflare says it has become the first internet infrastructure provider to block artificial intelligence companies from pillaging the content of websites to train their models, declaring the move a win for businesses and creators.
The AI boom is transforming the way people find information, with Google providing automated summaries at the top of searches, rather than directing users to websites, sparking a collapse in traffic for online publishers.
Others are bypassing Google entirely, relying on OpenAI’s ChatGPT and other models to find information – again without ever having to go to a website that houses the source material – with some media companies saying it has heralded “AI armageddon”.
San Francisco-based Cloudflare, which underpins about 20 per cent of the internet, said it is now giving “power back to companies and content producers” by blocking AI crawlers.
It has shifted from an opt-out to opt-in model, which Cloudflare co-founder and chief executive Matthew Prince said gives web publishers more control and prevents AI companies from ripping off their content.
“If the internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow’s AI founders, and the future of the web itself,” Mr Prince said.
“Original content is what makes the internet one of the greatest inventions in the last century, and it’s essential that creators continue making it. AI crawlers have been scraping content without limits.
“Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant internet with a new model that works for everyone.”
Mr Prince said AI companies will now be required to obtain explicit permission from a website before scraping. “Upon sign-up with Cloudflare, every new domain will now be asked if they want to allow AI crawlers, giving customers the choice upfront to explicitly allow or deny AI crawlers access”.
“This significant shift means that every new domain starts with the default of control, and eliminates the need for web page owners to manually configure their settings to opt out. Customers can easily check their settings and enable crawling at any time if they want their content to be freely accessed.”
Roger Lynch, CEO of Conde Nast – which brands include Vogue, GQ and The New Yorker – praised Cloudflare’s move, saying it “sets a new standard for how content is respected online”.
“When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership,” Mr Lynch said.
“This is a critical step toward creating a fair value exchange on the internet that protects creators, supports quality journalism and holds AI companies accountable.”
Neil Vogel, CEO of Dotdash Meredith, publisher of People, Better Homes & Gardens and other titles, said Cloudflare’s move would help ensure media companies accessed fair compensation for their content.
“We have long said that AI platforms must fairly compensate publishers and creators to use our content. We can now limit access to our content to those AI partners willing to engage in fair arrangements,” Mr Neil Voge said. “We’re proud to support Cloudflare and look forward to using their tools to protect our content and the open web.”
Pinterest CEO Bill Ready said the move would also allow content creators to continue to grow their audiences.
“Creators and publishers around the world leverage Pinterest to expand their businesses, reach new audiences and directly measure their success. As AI continues to reshape the digital landscape, we are committed to building a healthy internet infrastructure where content is used for its intended purpose, so creators and publishers can thrive,” Mr Ready said.
Reddit co-founder and CEO Steve Huffman agreed. “The whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled, and Cloudflare’s efforts are a step in the right direction for everyone,” he said.
HubSpot CEO Yamini Rangan told this masthead last month that AI isn’t just changing the rules; it’s completely “rewriting the playbook”.
Chatbots are replacing Google searches. People can get information directly from AI tools like ChatGPT in a heartbeat and in “zero clicks”. As a result, traffic that marketers and companies have relied on for the past 20 years is plummeting.
Google, which controls about 94 per cent of Australia’s search market, has even disrupted itself. It launched AI Overviews late last year in an effort to compete directly with ChatGPT. These AI-generated summaries appear at the top of Google searches, transforming the way people search and find information, negating the need to click on links to other websites.
“Almost everything in the way you go to market has changed with AI,” Ms Rangan said.
“Ten years ago, it was a lot about searching for information. Your customer would search, and they would land in maybe the top 10 blue links within Google, and we did everything possible to show up. Then your customers would click on it and come to your website.
“That is getting completely disrupted. In fact, Google has announced their AI overviews have reduced search traffic by 20 to 70 per cent. That disruption is going to continue.”
To join the conversation, please log in. Don't have an account? Register
Join the conversation, you are commenting as Logout