From 15 September 2026 Cloudflare changes its default settings for automated crawlers. If you run a site behind it, this is yours to decide, and you have until that date to opt out of the new defaults.
Three categories
- Search: stays allowed.
- Agent: blocked on pages that display ads.
- Training: blocked on pages that display ads.
A mixed-purpose crawler that combines Search with Training is handled by the most restrictive rule — it is caught by the Training block.
What is in your hands
Each category is set independently: allow, block on all pages, or block only on pages that display ads. The controls are available to all customers including the Free plan, and an existing customer can opt out of the new defaults at any time before 15 September through the zone’s security settings. Three content-use permission levels were added: immediate with no storage, reference for indexing and linking, and full for summarising and reproducing.
The numbers, and they are Cloudflare’s own
- 52% of crawler requests were for AI training as of June 2026, against 22% in spring 2025.
- Mixed-use crawlers are over 36% of crawler activity.
- More than half of internet traffic is no longer human.
- 20% of the web sits behind Cloudflare, and 36% of the most-visited sites.
What we could not verify
No statement from any major AI company confirms it has separated its crawler user-agents before the deadline, so assume neither compliance nor breach. Cloudflare’s blog frames the new default as applying to new domains, while the product documentation describes a changed default with an opt-out for existing customers; this piece follows the documentation. No figure is published for how many Kuwaiti sites sit behind the service.
Photo: An aisle of racks in a data centre. Photograph by PiDatacenters, CC BY-SA 4.0 via Wikimedia Commons.
Source: Cloudflare’s blog, developer changelog and product documentation.
Kuwait One, once a week.
The most useful stories we published this week, in your inbox.
No spam. Unsubscribe anytime.
