AI crawlers are not uniform traffic: how websites can determine access scopes

Websites want to be discovered by users without having their content scraped without restriction. These two goals do not conflict; the issue is that once all automated visits are labeled as "AI crawlers," the choice often reduces to either full openness or total blocking.

A more useful distinction asks whether a given visit aims to build a search index, train a model, or complete a task on behalf of a user. Different purposes involve distinct returns, burdens, and required controls. This article examines public changes from Cloudflare to discuss how content sites and product homepages can make this assessment. Fact-checking is current as of 2026-9-5 and does not involve modifying any website's actual configuration.

During review on 2026-9-8, Cloudflare's announcements and configuration documentation still listed 9-15 as a future default adjustment. This article retains the initial publication timeframe and contemporary context; whether changes take effect as planned upon arrival of those dates requires re-reading official documentation and cannot be preemptively declared here.

Shifting from an AI Switch to Access Purpose

In its 2026-7 announcement, Cloudflare further categorizes AI-related traffic into Search, Agent, and Training. Search is used to collect or index content for subsequent queries; Agent focuses on executing immediate tasks on behalf of users; Training is used for model training or fine-tuning. This represents Cloudflare's product taxonomy rather than a universally enforced standard across the internet.

This distinction deserves attention because a single public document may serve multiple roles: a search system reads it to help others find answers later, an assistant reads it now because a user is actively debugging an API, and training collection may be entirely unrelated to any current visitor's query.

For product homepages, the second type of visit is not inherently valueless. If a developer asks an assistant about the meaning of a parameter, whether the assistant can access version-accurate documentation affects the user's ability to proceed with integration. This value does not necessarily manifest immediately as a browser click.

However, this does not imply that all agents should be granted access. Whether a visit aligns with permitted uses, requires authentication, or imposes abnormal burdens still requires individual evaluation. Treating all traffic with "AI" in its name as a customer acquisition opportunity is just as crude as treating all of it as valueless traffic.

The 9-15 Schedule Cannot Be Stated as Active for All Websites

Cloudflare's configuration documentation states plans to adjust certain default behaviors on 2026-9-15: for newly onboarded domains, training and agent traffic are blocked by default while search is retained, alongside updates to how mixed-purpose crawlers that perform both search and training interact with training block policies.

This statement contains several non-negotiable caveats: the planned date, newly onboarded domains, applicable pages, and rules for mixed-purpose visits. At the time of review, the date remains 9-5, and it cannot be rephrased as "Cloudflare has blocked AI access for all websites." A website's ultimate behavior depends on actual settings and subsequent product updates.

If you use related services, what is worth checking is not how strong a headline a summary article uses, but the current account policies, page categories, and real access logs. A site-wide switch may affect multiple page purposes simultaneously, and a visitor classified as mixed-purpose cannot be assumed allowed based solely on its name.

Our assessment is that such changes should trigger a scope review rather than an emotional site-wide toggle. First list the content intended for discovery, then check whether current policies conflict with that goal. This article does not recommend copying any configuration, nor does it imply we have audited your account.

Robots.txt, Usage Preferences, and Actual Blocking Operate at Different Layers

RFC 9309 explicitly states that robots exclusion rules are not access authorization mechanisms. They instruct compliant crawlers on how to access a site, but cannot replace login credentials, permission checks, or server-side access control.

This means that documents requiring confidentiality should not rely solely on robots.txt for concealment. Conversely, publicly intended explanations may still be blocked by other network or application rules even if robots.txt permits crawling. Determining accessibility requires checking actual responses rather than relying on a single file.

Cloudflare's corresponding announcement also introduces content usage preference signals currently under test, noting that expressing preferences does not directly trigger blocks. Website operators should separately log "what preference I expressed," "what rules the service enforced," and "what actions visitors actually took." Submitting a signal does not guarantee that content will no longer be trained on or reproduced.

This article discusses technical and product scopes and does not treat access settings as a complete determination of content rights or usage licenses. Private information requiring restriction must be protected by proper identity and permission mechanisms, while public materials require clear versioning, attribution, and scope of use.

Being Read Does Not Equal Being Cited

Google's AI search feature documentation continues to emphasize foundational SEO: crawlability, discoverability via internal links, core content presented as text, and structured data aligned with page body text. To qualify as a supporting link for relevant AI search features, pages must meet indexing and snippet display eligibility, though meeting requirements guarantees neither crawling, indexing, nor display.

Consequently, a crawler request appearing in server logs indicates only that an access event occurred. It does not prove the page entered a search index, let alone that a specific AI response utilized the content. A brand mention in a response does not necessarily mean the source list cited that page.

For content teams, this distinction alters daily evaluation. When an article is newly published, teams can first verify public access, links, and content correctness, and subsequently review search system indexing and display. Using "whether AI cited it" as the sole acceptance criterion from the outset risks premature failure determinations or mistaking random mentions for long-term growth.

Callers face a similar issue: search matching, body reading, and conclusion verification are distinct steps. In-house agent search tool selection guides explain this distinction from the user's perspective, whereas this article examines access and distribution from the content owner's perspective, avoiding merging the two into a single metric.

Decide Based on Page Purpose, Not Attitude Toward AI

We recommend categorizing pages before discussing access purposes. Public product documentation typically focuses on helping users understand and complete integration; original research articles may prioritize attribution, citation, and ongoing maintenance; and post-login customer data must remain restricted by user permissions rather than exposed to increase visibility.

For each page category, four specific questions can be asked:

1. Who is intended to read it in what context: prospective users discovering the product, or existing users troubleshooting?
2. Can automated access help complete this task, and what observable evidence supports this?
3. Which access patterns fall outside expected ranges, and do they require preference declarations, traffic controls, or strict authentication?
4. If policies are adjusted, how can you confirm target users are not adversely affected and that original settings can be restored?

This does not require writing complex policies for every individual crawler, but rather ensuring rule modifications have explicit rationales. For instance, if API documentation is intended to help users troubleshoot, verify whether normal user-initiated reads successfully return body content; if data is restricted to logged-in users, test whether unauthorized requests are properly rejected. Success criteria for each differ entirely.

Rule changes can also alter data samples. If a public source becomes temporarily inaccessible, an observation system receives less content, but this does not equate to a genuine decline in external discussion. When conducting cross-source analysis, coverage gaps should be preserved rather than treating missing data as zero. In-house multi-platform reputation monitoring blueprints discuss separating collection status from business conclusions as a methodological note rather than observed customer failures.

Evaluating Returns and Avoiding Disruption to Normal Use

Following policy adjustments, track metrics across consistent observation windows: automated traffic volume and error rates, critical page accessibility, search impressions and clicks, verifiable citation links, and user conversion behaviors leading to documentation or integration completion. Avoid aggregating these figures into an ambiguous "AI traffic" metric.

Certain returns may remain temporarily un-attributable. For example, if a user learns about a product through an assistant before visiting the homepage directly, standard source tracking may fail to capture the complete journey. The correct approach is to preserve this unknown portion rather than supplying estimated figures to prove policy effectiveness.

AI access management is growing more granular, and website governance should follow suit. First clarify which content serves whom, then verify access purposes, actual rules, and usage outcomes. For homepages requiring discovery, the goal is not maximizing log entries from automated bots; for protected content, it is not ensuring all automated requests fail. The objective is to provide appropriate individuals and tools with reliable information sufficient to complete their tasks within permitted boundaries.