How to Block AI Scrapers: robots.txt, Rate Limits, and Access Control

Direct answer: You can make private content reliably unavailable to unauthenticated scrapers, and you can make bulk collection of public content slower, costlier, and easier to detect. You cannot guarantee that material visible to any ordinary browser will never be copied. Use robots.txt to publish preferences to cooperative crawlers, verified bot and web application firewall rules to enforce request policy, rate limits to control resource use, authentication and authorization to protect private content, logs to preserve evidence, and legal terms to state permitted use. No one layer does all six jobs.

Evidence status: This draft is documentation-backed. TechGeeks reviewed the cited standards, crawler documentation, CDN controls, and legal primary sources on August 15, 2026. No TechGeeks production site, CDN account, crawler traffic set, legal dispute, or rate-limit load test was provided or tested. The implementation and acceptance tests below are recommendations and planned lab work, not reported results.

The Short Version

• Public and discoverable: Put crawler-specific preferences in /robots.txt, then monitor whether named crawlers fetch and follow them.
• Public but resource-intensive: Add edge bot classification and rate limits. Start in log-only mode, verify identity, and preserve search, uptime monitoring, accessibility, and partner traffic.
• Private, paid, personal, or confidential: Require real authentication and per-request authorization. Do not publish a secret path and hope robots rules hide it.
• Need accountability: Keep privacy-conscious edge and origin logs, a versioned policy record, and clear legal terms. These create evidence; they do not retroactively remove copies already collected.
• Need an absolute guarantee: Do not serve the material to the public internet. Even a good anti-bot stack cannot stop an authorized human, compromised account, residential proxy, screenshot, browser automation, or downstream copy from reproducing public content.

1

robots.txtPublishes crawl preferences for cooperative automated clients.Preference

2

TermsStates permitted use, licensing, notice, and dispute terms.Legal notice

3

Rate limitsCaps request volume or cost by a chosen identity and window.Resource control

4

Verified edge rulesBlocks or challenges requests using identity and behavior signals.Active enforcement

5

AuthenticationRequires an approved identity and authorization before content is served.Access boundary

What robots.txt can prove

It can prove what preference the server published at a recorded time and which path rules a compliant parser should apply. It cannot prove a requester read the file, obeyed it, or deleted older copies.

What an edge block can prove

It can prove that requests matching the tested rule were denied at that edge. It cannot prove that every disguised or distributed request matched, or that another source did not supply the content.

What authentication can prove

It can prove that the tested unauthenticated path did not receive protected content and that an authorized identity did. It cannot prevent an authorized account from copying what it can view.

What logs and terms add

Logs can connect a request, response, policy version, and timestamp. Terms can state restrictions and a licensing path. Their evidentiary and legal effect depends on integrity, notice, assent, jurisdiction, and specific facts.

How to define a rate limit

Identity: choose a counting key such as an IP address, account, API key, cookie, verified bot, or a combination, then scope the rule to the relevant host, path, or action. Window: document the burst allowance, sustained counting period, reset behavior, and any Retry-After response. Failure modes: shared addresses can penalize legitimate users, distributed addresses or account rotation can evade a limit, and automatic retries can add load. Start with a canary and keep a tested rollback.

Limits of verified-bot identity

A claimed user-agent name is not verified identity. Depending on the bot, verification may use current vendor-published IP ranges, matching forward and reverse DNS, cryptographic signatures, or a maintained edge-provider classification. A match supports only that request and method at that time; unknown, relayed, compromised, user-initiated, or changed identities may remain. Recheck official identity documentation before enforcement.

Hard limit: Once the same content is delivered to ordinary anonymous browsers, the server cannot know with certainty whether a human, automation, a compromised device, or a downstream service will copy it.

First Decide What You Are Trying To Stop

"Stop AI scrapers" is too broad to configure. A publisher may want search visibility but no model training, user-requested AI answers but no archive crawl, a slower crawl rate, protection against infrastructure cost, or confidentiality for paid and private material. Those are different policies.

Goal	Primary control	Supporting controls	Honest promise
Opt out of a cooperative training crawl	Crawler-specific robots.txt group	Versioned terms and log review	The preference is published; compliant named crawlers should follow it.
Keep search discovery while limiting training	Separate search and training tokens	Verify current vendor documentation	Possible only where the vendor separates those purposes.
Reduce scraping load	Rate limits and caching	Bot classification, cost-based path rules, alerts	Request cost is bounded for identities and paths the rule can recognize.
Block known automated clients	Verified bot or WAF rule	Published IP ranges, reverse DNS, signatures, behavior signals	Matched requests are denied; unknown and disguised automation may remain.
Protect confidential content	Authentication plus authorization	Origin isolation, private caching, expiring links, audit logs	Unauthenticated and unauthorized requests do not receive the protected representation.
Preserve a legal position	Terms drafted for the business and jurisdiction	Policy versioning, notice records, crawl logs, licensing contact	The operator can document notice and conduct; enforceability is fact-specific.

What robots.txt Actually Does

Documented fact: RFC 9309 is the IETF Standards Track specification for the Robots Exclusion Protocol. A crawler retrieves a UTF-8 plain-text file at the lowercase top-level path /robots.txt. Product-token matching is case-insensitive. Path matching should be case-sensitive, starts at the beginning of the path, and uses the most specific matching rule. If equally specific Allow and Disallow rules conflict, Allow should win.

The scope is the exact scheme, authority, and port. A file at https://example.com/robots.txt does not automatically govern http://example.com, https://www.example.com, another subdomain, or a nonstandard port. This is a common source of partial coverage.

The RFC also defines failure behavior. A successfully downloaded file must be followed by a compliant crawler. A 4xx response makes the file "unavailable," after which the crawler may access resources. A server or network failure makes it "unreachable," and the crawler must initially assume complete disallow. Crawlers should not normally use a cached copy for more than 24 hours unless the file is unreachable. Vendor behavior can be more detailed: Google documents its own cache and extended error handling, so do not deliberately break /robots.txt as a blocking strategy.

Security boundary: RFC 9309 explicitly says the protocol is not a substitute for valid content security. The file is public, and listing a path can make that path easier to discover. A Disallow line does not make a URL private, return a denial response, remove an indexed URL, or authenticate the requester.

A purpose-specific example

The following example expresses a policy, not a firewall rule. It allows named search crawlers while asking named training crawlers not to fetch anything. The tokens and their purposes must be rechecked against current vendor documentation before use.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Google-Extended
Disallow: /

OpenAI currently documents GPTBot for potential foundation-model training and OAI-SearchBot for ChatGPT search. It says those controls are independent and that search adjustment can take about 24 hours. It separately documents ChatGPT-User for user-initiated actions and says robots rules may not apply. Google documents Google-Extended as a control token, not a separate HTTP user agent, for specified Gemini training and grounding uses. It is therefore not a request identity that a WAF can match directly. Blocking it does not itself block Google Search. Anthropic documents separate ClaudeBot, Claude-SearchBot, and Claude-User purposes.

Those names are product policy, not permanent Internet standards. The 2026 IAB AI-CONTROL workshop report records substantial variation in how AI crawlers interpret preference mechanisms and separate crawl-time, training-time, and user-agent use cases. Maintain a crawler register instead of copying an old block list and forgetting it.

Recommendation: keep the policy small and repeat common restrictions in each named group. Under RFC 9309, a crawler that matches a specifically named group does not also inherit rules from the wildcard group. Test the public file from outside your network after every change.

User-Agent Names Are Not Bot Identity

Documented fact: An HTTP User-Agent value is supplied by the requester and can be spoofed. Google and Bing both warn operators to verify crawler traffic rather than trusting the string alone. Google publishes crawler IP ranges and reverse-DNS patterns. Bing provides an IP verification tool. OpenAI and Perplexity publish JSON IP ranges for their named bots. Anthropic publishes a crawler IP list.

Recommendation: Treat a user-agent match as a label, not an identity credential. For a known crawler, combine the documented token with a current official IP range, forward-confirmed reverse DNS where the operator supports it, or a CDN's verified-bot classification. Cache successful DNS verification for a limited period rather than performing multiple DNS lookups in the request path. Refresh published IP data safely and fail to the policy you chose when that refresh fails.

Cryptographic request identity is emerging. Cloudflare's Web Bot Auth documentation describes HTTP message signatures and a public key directory for verified bots and agents. Cloudflare also documents implementation limits and reliance on evolving IETF drafts. This is useful progress, but it is not yet a reason to assume every crawler signs requests or every origin can verify them. Keep IP and behavior fallbacks until the traffic you depend on proves otherwise.

What Edge Bot Controls Enforce

A CDN or WAF can actively block, challenge, log, or rate-limit a request before it reaches the application. That is a technical control, unlike robots.txt. Its weakness is classification: a rule can enforce only on traffic it matches.

Cloudflare currently separates AI traffic into Search, Agent, and Training behaviors and offers policies for those uses. As of this fact check, its documentation also announces changed defaults for new domains on September 15, 2026, and deprecation of the legacy Block AI Bots option on that date. This is exactly the kind of volatile product behavior that must be reopened on publication day, along with plan availability and rule order.

Cloudflare's own 2025 report described traffic it attributed to an undeclared crawler rotating user agents, IP addresses, and source networks after declared identities were blocked. Perplexity's July 2026 help documentation says it now respects robots.txt, disabled an earlier blocked-URL summary behavior, and updated third-party crawler agreements. TechGeeks did not reproduce either account. The operational lesson is narrower than the dispute: a named-user-agent block is not complete coverage, vendor behavior changes, and an old incident report is not publication-day crawler policy.

A safer rollout pattern

1. Inventory search crawlers, AI search, user-initiated agents, uptime monitors, payment callbacks, feed readers, accessibility services, partners, and internal automation.
2. Define allow, rate-limit, challenge, and block outcomes by purpose and path. Protect login, search, export, archive, and expensive rendering routes separately.
3. Deploy the proposed classifier in log-only mode. Compare edge classification with official IP data and origin behavior.
4. Canary enforcement on a low-risk path. Test verified allowed bots, unverified spoofed user agents, ordinary browsers, mobile clients, and monitoring.
5. Expand gradually, alert on denied legitimate traffic, and retain a tested skip rule or rollback.
6. Block direct access to the origin. A public origin address can bypass a perfect edge policy.

Recommendation: Do not automatically allow a request merely because it claims a famous crawler name. Also do not block all automation indiscriminately. Search indexing, link previews, security scans, feeds, and monitoring may be valuable. Policy should follow verified purpose and business need.

Authentication Is The Boundary For Private Content

If content must be private, the application must decline to serve it until a requester proves an allowed identity and passes authorization for that object. RFC 9110 defines the HTTP authentication framework: a 401 Unauthorized response challenges for valid credentials, while 403 Forbidden means the server understood the request but refuses it. The exact choice depends on whether authentication could change the result and whether disclosing that distinction is appropriate.

Recommendation: Put member archives, paid downloads, account data, drafts, source documents, and administrative APIs behind authentication at the application or a trusted access proxy. Enforce authorization on every request, not just navigation. Use short-lived signed URLs only where the leakage and forwarding risk is acceptable. Prevent public CDN caching of private responses. Remove alternate origins, object-storage URLs, RSS feeds, print views, JSON endpoints, and previews that expose the same content without the gate.

Authentication is much stronger than bot detection because it asks, "Is this requester allowed to receive this resource?" rather than, "Does this request look automated?" It still cannot prevent an authorized customer, compromised account, browser extension, or endpoint from copying what that account can see. Use least privilege, download limits, account alerts, and contractual controls when insider or account misuse matters.

Rate Limits Control Cost, Not Intent

RFC 6585 defines 429 Too Many Requests for rate limiting and allows a Retry-After header. It deliberately does not define how the server identifies a user or counts requests. A CDN may count by IP, verified bot, account, API key, cookie, path, host, or a combination.

Recommendation: Limit the scarce resource, not every page equally. A static cached article may tolerate more anonymous requests than site search, PDF rendering, archive enumeration, GraphQL, image transformation, or an export endpoint. Use a burst allowance for normal browsers, a sustained limit for long crawls, and stricter authenticated quotas for expensive account actions. Return a stable 429 response and a practical retry interval where the client can recover.

An IP-only limit can punish offices, schools, carrier-grade NAT users, or privacy relays that share an address. It can also be bypassed with distributed addresses. An account-only limit does nothing for anonymous scraping and can be evaded through account creation. A user-agent-only limit is trivial to rename. Combine signals, but keep the decision explainable enough to diagnose false positives.

Logging Turns A Rule Into Evidence

Without logs, an operator cannot distinguish a crawler that stopped, a crawler that changed identity, a cached response, an origin bypass, a false positive, or an unrelated traffic drop. Cloudflare's HTTP request dataset, for example, documents request, response, bot score, detection source, verified-bot, cache, and security fields, with some fields dependent on product entitlement. The OWASP Logging Cheat Sheet emphasizes that event data from another trust zone is untrusted and that credentials, access tokens, session identifiers, and unnecessary personal data should not be written directly to logs.

Recommended minimum event: UTC timestamp, edge request ID, host, normalized path class, method, response status, bytes, cache result, latency, source-network data appropriate to the privacy policy, presented user agent, verified identity or classification source, rule ID and action, origin status, and authenticated account ID only when needed and pseudonymized where practical. Record the policy version that produced the decision.

Protect logs from alteration and unauthorized access, define retention, and alert if logging stops. Do not put query-string tokens, cookies, authorization headers, passwords, full form bodies, or private document text into a scraper investigation log. A large log archive can become a more sensitive copy of the site than the site itself.

What Legal Terms And Copyright Can Add

This section is general information, not legal advice. Terms can state that automated collection, model training, resale, circumvention, or use above a rate is prohibited or requires a license. They can name an effective date, identify the rights holder, provide a contact, and preserve evidence of the policy presented when access occurred. They do not make the server reject a request.

Enforceability is not automatic. It can depend on jurisdiction, ownership, the type of work or data, contract formation, notice, assent, the exact conduct, technical access controls, damages, privacy law, competition law, and available claims. A banner, footer link, robots.txt rule, cease-and-desist letter, signed license, and login click-through do not all create the same record.

Two U.S. decisions illustrate why broad claims are unsafe. In Van Buren v. United States, the Supreme Court interpreted the Computer Fraud and Abuse Act's "exceeds authorized access" language around access gates rather than improper purpose alone. In the Ninth Circuit's hiQ Labs v. LinkedIn preliminary-injunction opinion, the panel found a serious question whether the CFAA's "without authorization" concept applied to public profiles that required no prior authorization. That opinion did not declare all scraping lawful or resolve every contract, copyright, privacy, trespass, or state-law theory.

Copyright analysis is also fact-specific. The U.S. Copyright Office's May 2025 Part 3 generative AI training report describes copying and fair-use questions, licensing markets, and the absence of a mechanical fair-use formula. It was still labeled a pre-publication version in the source reviewed for this draft. It does not turn robots.txt into a universal license switch or decide a particular publisher's dispute.

Recommendation: Have qualified counsel align the site's terms, click-through or account agreement, licensing process, robots policy, privacy notice, and incident evidence with the jurisdictions and content involved. Keep machine-readable and human-readable policies consistent. Do not threaten claims the facts do not support, and do not assume a technical bypass automatically establishes a particular legal violation.

Recommended Implementation Runbook

Unperformed lab plan: The following sequence has not been run by TechGeeks for this article. Apply it first to a staging clone or low-risk canary behind the same edge and origin pattern as production.

1. Capture the starting state. Export DNS, CDN, WAF, rate-limit, reverse-proxy, origin firewall, authentication, cache, and robots.txt configuration. Record owners and a known-good rollback.
2. Inventory content paths. Label each as public-searchable, public-no-training preference, public-rate-controlled, authenticated, paid, or administrative. Find duplicate delivery through APIs, feeds, media URLs, archives, print views, and storage origins.
3. Build the crawler register. Record purpose, official documentation URL, product token, user-agent examples, official IP or signature method, allowed paths, expected rate, and business owner. Give every entry a review date.
4. Publish a small robots policy. Serve a 200 response with UTF-8 text/plain at every relevant authority. Avoid advertising secret path names. Validate longest-match behavior and product-specific groups.
5. Establish a log baseline. Measure request rate, path distribution, origin cost, cache ratio, error rate, verified bots, unknown automation, and human conversion before blocking.
6. Stage edge rules in log mode. Verify declared crawlers with official identity data. Separate search, agents, training, monitoring, partners, and unknown automation.
7. Canary rate limits and blocks. Apply them first to expensive or abusive paths. Preserve a bypass for a named operator and test that the bypass cannot reach the origin from the public internet.
8. Gate private content. Require authentication and object-level authorization. Confirm logged-out, wrong-account, expired-link, direct-origin, and cached-response failure paths.
9. Publish aligned terms. Record the version, effective time, presentation method, licensing route, and owner. Obtain legal review where the restrictions matter commercially.
10. Expand only after acceptance. Compare edge and origin logs, user-visible behavior, search tools, alerts, and rollback. Keep unknown automation under observation instead of pretending it has disappeared.

Verification Checklist

Run these checks from an ordinary external network, an authenticated test account, a wrong-account test, and an authorized administrative path. Use only systems you own or are authorized to test.

• /robots.txt returns 200, the intended Content-Type, and the reviewed policy on every scheme, host, and port that serves content.
• A standards-aware tester resolves representative paths to the intended longest-match rule for each named token.
• A request merely claiming Googlebot, GPTBot, or another allowed name is not treated as verified identity.
• Verified allowed search traffic reaches allowed public pages, while a matched blocked training policy receives the intended edge action.
• Ordinary browsers, accessibility workflows, uptime checks, feeds, payment callbacks, and partner integrations still work.
• The first request above a canary threshold receives the expected 429 and, where appropriate, Retry-After; service recovers when the window expires.
• An unauthenticated private request receives no protected body. A wrong account cannot read another account's object. An authorized test account can complete the intended workflow.
• The origin is unreachable except through approved infrastructure. Alternate storage, API, print, feed, preview, and media paths do not expose the protected representation.
• Edge and origin logs share useful UTC timestamps and request IDs, record the rule and outcome, and omit credentials and private content.
• The alert reaches an owner, identifies the affected rule and path class, and includes the tested rollback procedure.

A simple curl -A GPTBot request can confirm how your server responds to that presented string. It cannot prove the request came from OpenAI or that a vendor will process robots.txt as expected. Identity verification and crawler-side testing are separate checks.

Failure And Recovery

Legitimate search or monitoring was blocked

Disable or skip the newest terminating edge rule, not authentication for private content. Confirm the requester against its current official identity method, then restore the known-good rule set. Check WAF events and origin logs for the first denied request. Restore the known-good robots.txt file if it changed, but expect compliant crawlers to retain a cached copy for a period. Use the relevant webmaster inspection tool after the live file and page response are correct.

The rate limit harmed real users

Roll back the canary threshold or change the counting key while preserving logs. Look for shared NAT, IPv6 aggregation, privacy relay, retry storms, or a frontend that makes many hidden requests. Restore service first, then tune bursts and sustained windows against measured normal traffic. Do not solve a bad anonymous limit by exempting every request with a browser-like user agent.

Private content was accidentally public

Reinstate authentication or remove public routing immediately. Purge public caches where supported, disable leaked signed URLs, rotate exposed tokens, preserve edge and origin logs, and identify every alternate copy. Treat this as an exposure incident, not a robots.txt problem. Notification, takedown, customer, insurer, and legal duties depend on the data and jurisdiction.

A robots change removed content from discovery

Restore the intended specific search-crawler group and verify that a generic or duplicated group is not overriding the policy. Check page-level noindex and X-Robots-Tag separately. Google warns that a crawl-blocked URL can still appear without content details and that a crawler cannot see a page-level noindex when robots rules prevent the fetch. Correct the live responses first, then request recrawling through the official search tool where appropriate.

Evidence Limits

• A correct robots.txt response proves the published preference, not compliance by every crawler.
• A drop in one named user agent does not prove collection stopped; identity may change, traffic may come through a user-initiated agent, or content may come from another index.
• A WAF denial proves that matching requests were blocked at that edge, not that every route to the content was closed.
• A successful authentication test proves only the tested identities, objects, caches, and paths. It does not prove an authorized recipient cannot copy content.
• A 429 test proves the configured counter acted in the tested window. It does not prove resistance to distributed requests or account creation.
• Logs show requests and responses visible to the logging points. They usually cannot prove how a downstream party stored, licensed, trained on, summarized, or deleted content.
• The cited U.S. legal sources do not determine another jurisdiction or a specific dispute. No legal conclusion was tested for TechGeeks.
• Cloudflare's reports and product classifications are Cloudflare evidence. TechGeeks did not inspect its underlying cross-customer telemetry or reproduce the historical Perplexity findings.

Publication-Day Rechecks

• Reopen RFC 9309 and its RFC Editor status and errata pages. Confirm no verified erratum changes the parsing or failure guidance used here.
• Reopen the OpenAI, Google, Anthropic, Perplexity, and Bing crawler pages. Confirm product tokens, purposes, user-initiated behavior, IP feeds, signatures, and stated update delays.
• Reopen Cloudflare's Block AI Bots, verified bots, Web Bot Auth, AI Crawl Control, rate limiting, and logging pages. Confirm the September 15, 2026 transition, plan availability, navigation, rule order, and field names.
• Check whether the U.S. Copyright Office replaced the Part 3 pre-publication report with a final report or issued substantive corrections.
• Check the current status and later history of any legal authority retained in the article, and have counsel review any site-specific legal language.
• Validate every external reference and every published TechGeeks cross-link. Confirm that no draft-only TechGeeks URL was added.
• Confirm the article remains a WordPress draft with Gutenberg blocks, one responsive no-JavaScript interactive model, no in-content H1, no downloadable quick card, and no affiliate link or disclosure.

Related TechGeeks Reading

• Network Security Field Notes: Start Here
• AI Workflow Notes: Start Here
• What Is the Safest Way to Expose One Self-Hosted App Publicly?
• Reverse Proxy Setup Without Security Mistakes
• What Should You Monitor in a Homelab?

References

• RFC 9309: Robots Exclusion Protocol
• RFC 9969: IAB AI-CONTROL Workshop Report
• RFC 9110: HTTP Semantics
• RFC 6585: Additional HTTP Status Codes
• OpenAI: Overview of OpenAI Crawlers
• Google: Common Crawlers and Google-Extended
• Google: Googlebot Access and Verification
• Google: Block Search Indexing With noindex
• Anthropic: Web Crawlers and Site Controls
• Perplexity: How It Follows robots.txt
• Bing Webmaster Tools: Verify Bingbot
• Cloudflare: Block AI Bots
• Cloudflare: Verified Bots
• Cloudflare: Web Bot Auth
• Cloudflare: Rate Limiting Rules
• Cloudflare: HTTP Request Log Fields
• Cloudflare: 2025 Perplexity Crawler Findings
• OWASP: Logging Cheat Sheet
• U.S. Copyright Office: Copyright and Artificial Intelligence, Part 3
• U.S. Supreme Court: Van Buren v. United States
• U.S. Court of Appeals for the Ninth Circuit: hiQ Labs v. LinkedIn
• Hackaday: Perplexity v. Cloudflare Context