Cloudflare DNS to Block AI Bots
This is why I love using Cloudflare for DNS. They're on our side.
Yes, I love this company and we use a lot.
Just adding some links (under correct post)
Pay Per Crawl - This is a new initiative from Cloudflare to let content creators set their rates and get paid by AI services that want to crawl your content. If you want to sign up for Pay Per Crawl, click the link. Publishers can block all AI crawlers, allow specific ones, charge for access, or grant free access — full control over how their content is accessed. AI crawler owners will be able to use Pay Per Crawl to register, see pricing, and choose to pay or not. Pricing will set by both publishers, who can set rates, and AI companies, who can choose whether to access webpages at those rates.
Many people are mad that AI is “stealing” content, but nobody wants to talk about the real issue: you outsourced your business model to Google traffic, ads, and SEO.
Blocking bots won’t fix that. Traffic isn’t a currency. Likes aren’t revenue. If your model collapses because fewer people click ads, the model was broken to begin with. AI doesn’t replace your product—it just skips the middlemen.
If you want sustainable income, own your audience, own your value, and stop pretending someone else owes you a living.
Andre Daus I can tell you're very passionate about this issue.
You're so passionate that you've made a lot of assumptions about someone you never met and you're being quite rude in the way you get your point across.
Worse, you entirely missed the point.
The problem that Coudflare is trying to address, and the reason I prefer to block known AI bots, has nothing to do with SEO or trying to get traffic.
These bots are, as stated in the article that I'm guessing you didn't bother to read, causing stress on servers like a denial of service attack.
Whether you use AI or not is irrelevant. It's using your site. It may be causing performance problems for you and your audience, whether you rely on SEO or not.
So please stop lecturing me and pretending that your view is the only one that exists. This issue isn't about getting traffic.
William Beem I dont think Andre argue with you or something but he is just highlighting other point of view. Its not who is right here...
I think "blocking site content" for AI bots will be a good marketing product from CF where they will make some money but I am not sure how they want to block some scapers, automations or the site content being sent to AI once the copy was already obtained (by the middle man content copycat who paid one-time for scraping etc) - Let`s see..
ps. AI Act is coming in EU but anybody will follow these rules? Hmm...
Mat Perhaps you are right. When someone replies to me and uses the word "you" as he did, I tend to take that at face value and find it insulting.
William Beem you are really awesome person and I dont think anybody wants to prove you are wrong or something rather then just giving some extra questions and approach.
Dont stop sharing your awesome knowledge and your perspective - I will still love you haha...
ps. if CF will block AI Bots then we will open a project where we will be scraping content and feeding AI (from our DB) without CF knowledge. Then we can sell it. To be honest... I dont think we will be able to stop AI madness just because the Big Guys already has access to the content and probably train their AI for the last couple "days"...
ps2. dont forget that when two experts meet there are at least three opinions and nobody is right or wrong (here we have 8k+ Ninjas and this is our Flower Power).
Mat Thank you.
William Beem thanks for the clarification. You’re right—the server load and bot behavior angle is a separate and serious issue, and I agree that’s worth addressing regardless of your monetization model.
My answer was aimed more at the broader discussion I see again and again—not just here, but in many communities where I engage with this topic. Many creators are understandably frustrated that AI models are using their content without permission or compensation. At the same time, though, I often hear from the same people that they want their content to be found by AI systems.
The contradiction is striking: you can’t have it both ways. If you block access entirely, you can’t expect to show up in AI-generated answers—just like blocking Google’s crawler means you won’t show up in search results.
And whether we like it or not, AI-driven discovery may soon drive more traffic than traditional search engines ever did. If we want to be part of that ecosystem, we need to adapt—not by giving everything away blindly, but by rethinking how we deliver value, connect with audiences, and build resilience into our models.
So while I respect concerns about server strain (and agree that should be addressed), my real concern is how many people are reacting to AI by doubling down on dependency—hoping for compensation or protection from platforms, instead of building something more direct, more sustainable, and more independent.
If my tone came across as dismissive, that wasn’t the intent. I'm just challenging what I see as a bigger issue behind the immediate frustration.
Andre Daus Thank you. I appreciate the clarification and your intent.