On September 15, 2026, Cloudflare changes how it handles AI bots by default, and one line in that change can cut a store off from Google Search. A zone that blocks AI training crawlers, including through the legacy one-click “Block AI bots” toggle, will also block Googlebot, Bingbot, and Applebot, because those crawlers collect content for search and for AI training in the same visit. Merchants have reported Cloudflare blocking Googlebot since early July, weeks ahead of the date. One documented case lost 42 indexed pages and 64% of search impressions in a week. Below: the mechanism, the evidence, and a 10-minute audit for any Cloudflare-fronted store.
What changes in Cloudflare on September 15, 2026?
Cloudflare retires its single “Block AI bots” switch and sorts AI bots into three behavior categories: Search, Agent, and Training. Search covers crawlers that index content to answer questions later. Agent covers bots acting in real time on a person’s behalf, such as the fetchers behind AI assistants. Training covers crawlers that collect content to train or fine-tune models (Cloudflare announcement, July 1, 2026).
On September 15 the new defaults take effect: Cloudflare blocks Training and Agent bots on pages that display ads, and Search stays allowed (Cloudflare changelog, July 2026). The defaults cover new domains onboarding to Cloudflare, new zones added by existing customers, and existing free-plan customers. Zones can opt out through Security Settings until September 15.
The second change shipping the same day is the trap. Cloudflare will judge mixed-purpose crawlers, the ones that crawl for search and training at once, by the most restrictive rule touching any of their behaviors. The announcement names the affected crawlers: “multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked” for any customer with a Training block, whether set through the new controls or the legacy toggle (Cloudflare, July 2026).
Cloudflare shipped the one-click block in July 2024, and over one million sites had enabled it by mid-2025 (Cloudflare, 2025). Each of those zones now carries a setting that reaches further than it did on the day someone clicked it.
Why does blocking AI training block Googlebot?
Google crawls with one bot for search indexing and for its AI features. Gemini and AI Overviews draw on Googlebot’s crawl, and the only separate control Google offers is Google-Extended, a robots.txt token that limits AI training use without touching search. Cloudflare classifies bots by behavior rather than by name, so Googlebot sits in the Search bucket and the Training bucket at the same time. A rule against the Training behavior catches Googlebot even on a zone where Search is set to Allow.
Bingbot and Applebot carry the same dual classification, and each loss costs a store something different. Blocking Bingbot removes pages from Bing’s index, which also feeds Microsoft Copilot’s web answers and supplies results to smaller engines; B2B buyers skew toward corporate Windows environments where Edge, Bing, and Copilot are the defaults, so the channel runs larger than its market-share number suggests. Blocking Applebot removes content from Spotlight, Siri, and Safari search experiences on Apple devices (Apple, 2026). Apple offers the same split Google does: Applebot-Extended is the robots.txt token for opting out of Apple’s AI training while staying in its search surfaces.
Blocked means a 403. Cloudflare answers the crawler with HTTP 403 Forbidden, and Google treats that status code as an instruction, covered in its own section below.
Merchants are already seeing Googlebot blocked
Three public reports show the Training block catching verified Googlebot before September 15.
zoundroom.com (Cloudflare community, August 25, 2026). The owner set ai_training=block through Cloudflare’s Bot Management API on August 15, with Search and Agent blocking off. Google Search Console then reported 403s on 68 URLs between August 15 and 22, and 47 of them were canonical URLs in the sitemap. Indexed pages fell from 114 to 72. Impressions dropped 64%. The zone’s audit log shows Cloudflare created a system ruleset one second after the config change. On August 24 the owner set Training back to Allow, Cloudflare deleted the ruleset, and GSC live tests passed again. Bot Fight Mode was off and the zone had no custom WAF rules, which isolates the Training setting as the cause.
Home-Barista.com (Cloudflare community, August 22, 2026). Verified Googlebot requests began receiving 403s in early July, with Cloudflare Security Events attributing the action to “Blocked by ‘Block AI training crawlers’”, the legacy toggle. Allowing mixed-purpose crawlers stopped the blocking.
A Reddit report covered by Search Engine Journal (July 2026). A site owner found Googlebot and Bingbot receiving 403s on sitemap fetches whenever AI Training was set to Block, and the 403s stopped when the block came off. Google’s John Mueller responded in the thread and asked for details.
These are merchant reports with audit-log evidence; Cloudflare had not confirmed the pre-deadline behavior as of publication. The pattern holds across three independent zones, and the fix in each case was the same: allow Training for mixed-purpose crawlers.
Why a 403 is worse than a 429 or 503
A 403 tells Google the page is forbidden, and Google removes forbidden pages from its index. Google’s crawling documentation treats 4xx responses (other than 429) as a signal that the content should not appear in search, so pages returning 403 drop out of results. Recovery requires a fixed response, a passing live test, and a recrawl, which takes days to weeks. The zoundroom case lost 42 of 114 indexed pages in one week of 403s.
A 429 or 503 buys time. Google treats those codes as temporary and retries for about two days before slowing its crawl; past roughly two days of sustained 429 or 503 responses, Google starts dropping those URLs as well (Google Search Central). Google published a warning on this in February 2023, after CDNs began rate-limiting Googlebot with 4xx codes: throttle with 500, 503, or 429, and keep it short.
Use 429 or 503 for any crawler you want to slow down. A 403 tells Google to remove the page from search.
The 10-minute Cloudflare audit
Run these four checks on each zone (each domain configured in your Cloudflare account) this week. One person with dashboard access finishes a zone in about ten minutes. If you are not sure whether Cloudflare fronts your store, ask whoever manages your DNS or CDN; a cf-ray header in any page response from your site also answers the question.
1. Check the AI bot settings on the zone
Open Security → Settings and filter by Bot traffic. Two places matter: the legacy Block AI bots setting, which Cloudflare deprecates on September 15, and the newer Configure AI bot policies panel with the Search, Agent, and Training presets. A Training block in either place puts Googlebot, Bingbot, and Applebot in scope after September 15, and per the reports above, perhaps sooner.
2. Set a deliberate policy before the deadline
Pick one of three routes. The first is time-boxed: opt out of the new defaults in Security Settings, available until September 15, which preserves the current treatment of mixed-purpose crawlers on existing zones (Cloudflare docs). The second: set Training to Allow; in the zoundroom case Cloudflare deleted the blocking ruleset within a second of that change. The third keeps the protection without the collateral damage: block training bots by name with AI Crawl Control (GPTBot, CCBot, Bytespider), and add Google-Extended and Applebot-Extended to robots.txt to opt out of Google’s and Apple’s AI training without touching Googlebot or Applebot.
A caution from the evidence: allowing Search does not by itself protect Googlebot. The zoundroom zone had Search set to Allow through the whole incident, and the Training block caught Googlebot anyway. The settings that decide the outcome are Training and the mixed-purpose treatment.
3. Search the WAF events for the block
Open Security → Events, Cloudflare’s log of what its web application firewall (WAF) and bot rules blocked, and search the last 30 days for actions attributed to Blocked by “Block AI training crawlers” or to the AI bot policy rulesets. Filter for verified bots. A verified Googlebot or bingbot entry with a 403 action confirms the block on your zone. Export the affected URLs; you will need the list for reindexing requests.
4. Confirm crawl health in Search Console and Bing Webmaster Tools
In Search Console, open Settings → Crawl stats and look for a 4xx spike, then check the Page indexing report for “Blocked due to access forbidden (403)”. Run a live URL Inspection test on two or three canonical URLs, and do the same with URL Inspection in Bing Webmaster Tools. If you find blocks: fix the Cloudflare setting first, re-run the live tests, then request indexing on the affected canonical URLs. The zoundroom owner followed this sequence and live tests passed the same day.
What “pages that display ads” means for an online store
The September 15 defaults apply to pages Cloudflare’s automated classifier identifies as displaying ads. Cloudflare has not published how the classifier treats retail media placements, sponsored product units, or third-party promo scripts, all common on category and search pages in Adobe Commerce and Shopware stores. A store with no display advertising in the AdSense sense may still serve markup the classifier reads as ad inventory.
The ads qualifier offers less comfort than it appears to. A Training block applies per your configuration, and the legacy toggle supports blocking on all pages. The pre-deadline reports above came from zones where the block fired without any ads question in play. Treat the qualifier as scope reduction and verify with the WAF events check rather than assuming exemption.
Should a store block AI crawlers at all?
Blocking pure training crawlers is a defensible choice; blocking by behavior category is the wrong tool for it while Google runs one crawler for search and AI. Training-only bots such as GPTBot, CCBot, and Bytespider identify themselves and can be blocked by name with no search impact. Behavior-category blocking sweeps in Googlebot, and Google has given no signal it plans to split its crawler.
Think twice before blocking the Agent category as well. Agent bots include the fetchers behind AI shopping assistants, which act for a buyer researching a purchase; for a B2B merchant, an agent fetch can precede an RFQ. Cloudflare’s own crawl data shows training drives close to 80% of AI bot activity (Cloudflare, August 2025), so name-level blocking of training bots removes most of the crawl load without touching search or agent traffic.
Our position: treat any Cloudflare bot setting as an SEO setting. Route changes through whoever owns your organic channel and verify in Search Console within 48 hours of any change. The merchants in the cases above lost indexed pages because a security-side toggle never crossed the desk of anyone watching search.
Frequently Asked Questions
Got some questions? We’re here to answer. If you don’t see your question here, drop us a line with out Contact form.
Is Cloudflare blocking Googlebot?
Cloudflare blocks Googlebot on zones where a Training block applies to mixed-purpose crawlers. From September 15, 2026 this is documented behavior for any Training block, including the legacy “Block AI bots” toggle. Merchant reports show the same blocking on some zones since early July 2026.
Does the “Block AI bots” setting hurt SEO?
After September 15 it will block Googlebot, Bingbot, and Applebot, because Cloudflare will enforce it against mixed-purpose crawlers. Pages served a 403 drop out of Google’s index. Replace it with name-level blocking of training-only crawlers if you want to keep the protection.
How do I know if Googlebot is being blocked?
Check Security → Events in Cloudflare for verified-bot 403s, and Google Search Console’s Crawl stats and Page indexing reports for “Blocked due to access forbidden (403)”. A live URL Inspection test gives a real-time answer for any single URL.
What happens if I do nothing ?
Paid zones with a saved configuration and no Training block keep their current behavior. New domains, new zones added by existing customers, and free-plan zones inherit the new defaults. A zone with a Training block, old or new, starts blocking Googlebot on the pages the block covers.
Should I return a 403 to bots I want to slow down?
No. Google removes 403’d pages from its index. Throttle with 429 or 503 and keep it temporary; sustained 429 or 503 beyond about two days also drops URLs from the index (Google Search Central).