Blog › Cloudflare's New AI Bot Defaults from 15th September 2026
13 Sep 2026 · Cloudflare · 5 min read
Cloudflare's New AI Bot Defaults from 15th September 2026 - What You Need to Know
In just a couple of days, Cloudflare is rolling out new default bot settings that could affect which AI crawlers can reach your website. If you use Cloudflare and haven't specifically changed your bot management settings, it's worth taking 5 minutes to check what's about to happen.
What is Cloudflare changing?
On 15th September 2026, Cloudflare is updating its default bot classifications to split AI traffic into three separate categories rather than treating all bots as one group, which they are calling Content Independence Day. The three categories are:
- Search - crawlers that index your site so it appears in search results and AI answer engines
- Agent - automated tools acting on a user's behalf in real time, for example fetching a page because someone asked ChatGPT a question
- Training - crawlers collecting content to train or fine-tune AI models
The new defaults on 15th September will block Training and Agent bots by default on pages that display ads, while allowing Search bots through as before. This applies to all new domains onboarding to Cloudflare and to existing free-tier customers who haven't changed their settings manually before that date.
There's also an important change for what Cloudflare calls "mixed use" crawlers, which are bots that combine Search with Training in one. If you've opted to block Training crawlers, these mixed bots will also be blocked under the new rules, even if you previously allowed them. This could potentially lead to drops in visibility in AI over time.
Worth checking now: if your site runs any advertising, and you're on Cloudflare's free tier, some AI agent bots may be blocked automatically from 15th September 2026 unless you've changed your settings before then.
Who does this actually affect?
The short answer is, this affects any websites using Cloudflare who hasn't manually reviewed their bot settings. Here's a quick breakdown:
| Situation | What happens on 15th September 2026 |
|---|---|
| New domain, any Cloudflare plan | New defaults apply automatically |
| Existing free-tier customer, settings unchanged | New defaults apply automatically |
| Existing paid customer, settings unchanged | No change to existing behaviour |
| Any customer who has changed settings before 15th September | Your settings are kept as-is |
If you're a paid Cloudflare customer and haven't touched your bot settings, Cloudflare says it won't automatically change your existing setup. But it's still worth reviewing, because the new controls are more granular and useful than the old "block AI bots" toggle.
What about Search bots like Googlebot?
This is where it gets slightly complicated. Googlebot and Bingbot are officially classified as "multi-purpose" crawlers, meaning they're used for both Search indexing and, in some cases, AI training. Under the new rules, if you've previously enabled "Block AI Bots" and haven't changed anything, Googlebot could be potentially be caught by these new Training restrictions.
Cloudflare has said they'll notify customers before the change takes effect and give everyone the chance to update their settings. But given that most people don't read platform notification emails, it's better to check now rather than wonder why your site disappeared from Google after 15th September. SEO agencies won't be able to blame this one on an algorithm update! That's why it's a good idea for all SEO agencies to have a tool like UAWatch - it's cheap at only £10 per site per month, they get the peace of mind that they can come into the office the next day and see if anything has changed, so that they can take quick action to remedy the problem. Plus, we offer a discount if you purchase more than three subscriptions at once.
Find out overnight if any of your bots have changed access status.
UAWatch checks 124 bots each night including GPTBot, ClaudeBot, Googlebot and all the major AI crawlers. If something does change after 15th September, you'll get an email the next morning telling you exactly what shifted and why.
Start monitoring - £10/site/month →How to check your Cloudflare settings before 15th September 2026
Log into your Cloudflare dashboard and go to Security > Settings. Look for the bot management section. You should now see the new three-way toggle for Search, Agent, and Training. Check what's set for each and make sure it reflects what you actually want.
If you want AI crawlers like GPTBot and ClaudeBot to be able to reach your site (which you probably do if you want to appear in AI answers), make sure Search and Agent bots are allowed. If you want to block training crawlers but keep search working, set Training to block and Search to allow.
The key thing is to make a conscious choice rather than letting the default take effect without you knowing about it. Whatever you decide is fine, just make sure it's intentional.
What about the "mixed use" crawler problem?
If you previously used the "Block AI Bots" toggle and left it on, you may now find that major search crawlers get caught in the new Training block. Cloudflare's recommendation is to switch to the new granular controls, which let you block Training without accidentally blocking Search. This is a good time to do that review.
Why is Cloudflare doing this?
It's worth being clear that Cloudflare is trying to give site owners more control, not less. The motivation behind these changes is that a lot of website owners were unhappy with a binary "block all AI" or "allow all AI" choice. The new system lets you say yes to AI search visibility (which most sites benefit from) while saying no to your content being used for model training without compensation.
The ad-page rule in particular makes sense if you think about it. Pages with ads are ones where the site owner is trying to generate revenue from human visitors. Training crawlers consuming that content without driving any traffic back are essentially taking something valuable without giving anything in return. Cloudflare is trying to protect that relationship.
They've also announced something called Pay Per Use, where publishers can eventually be compensated when their content appears in AI answers. That's worth keeping an eye on if you run content-heavy sites.
What should SEO professionals and agencies do?
If you manage client sites, 15th September is a good reason to do a quick audit of which clients use Cloudflare and whether their bot settings have been reviewed. A site that gets Training crawlers accidentally blocked won't see any immediate search ranking drop, but over time it could mean the site stops appearing in AI-generated answers in tools like ChatGPT and Perplexity.
For clients on Cloudflare's free tier who haven't changed anything, check their settings before Monday. For paid clients, it's lower urgency but still worth doing a review of the new controls since they offer more precision than the old toggle.
Monitor the impact overnight, automatically.
UAWatch will catch any bot access changes caused by the new Cloudflare defaults and email you the morning after they happen. No need to manually check each client site after 15th September.
Start monitoring - £10/site/month →Written by Kelly Sheppard, The Structured Data Company. Sources: Cloudflare Blog, Cloudflare Press Release, 1 July 2026.