Industries · News Publishers
Blocking GPTBot while leaving Google open doesn’t hide you from AI. It just costs you traffic.
A common robots.txt config at US newspaper sites disallows AI crawlers by name while leaving the wildcard user-agent — including Google — fully open. Anything Google can read, an AI assistant can read too. The block doesn’t work as intended, and per a recent Rutgers/Wharton study, it correlates with lower traffic, not protection.
What we found across ~7,000 US newspaper sites
6,963
unique newspaper hosts audited
~30%
block AI bots while staying open to search (the anti-pattern)
~15%
not evaluable — fetch failed or blocks everyone
1,519
hosts with recent-evidence audit coverage (Tier 1)
Source: PO robots.txt audit of the “usNewspaperPublisherInventory” site list (6,963 unique included hosts), cross-referenced against Strategic Response of News Publishers to Generative AI — Hangcheng Zhao (Rutgers Business School) & Ron Berman (The Wharton School, University of Pennsylvania), April 15, 2026 (staggered / synthetic difference-in-differences).
The anti-pattern
A robots.txt that blocks GPTBot by name, then opens the door to everyone.
This is the config we see most often on sites in the anti-pattern cohort: a specific User-agent: GPTBot rule set to Disallow: /, immediately followed by a wide-open wildcard rule.
User-agent: GPTBot Disallow: / User-Agent: *
Visible to Google = visible to AI.
The wildcard rule (User-agent: *) covers every crawler that isn’t named above it — including Googlebot. Naming one AI bot and leaving the wildcard open doesn’t make the site invisible to AI; it just blocks one named crawler while every other path in (including AI systems that read what Google indexes) stays open. It likely isn’t achieving the intended goal.
Worse: per the paper below, large publishers that block this way see reducedwebsite traffic compared to publishers that don’t — the block correlates with a cost, not a protection.
AI anti-pattern prevalence
Roughly 3 in 10 newspaper sites run the detrimental config.
From our robots.txt audit of ≈7,000 US newspaper sites (6,963 unique hosts, the “usNewspaperPublisherInventory” universe). This split is our own separate robots.txt audit — the inventory spreadsheet itself carries no per-site AI-block classification.
Sample: 6,963 unique included newspaper hosts ( usNewspaperPublisherInventory.xlsx site inventory). The inventory itself provides the host list only — no robots.txt / AI-block classification was requested during that phase; the 30/55/15 split above comes from a separate, subsequent robots.txt audit against this host list.
What the paper found
Blocking GenAI bots correlates with less traffic, not more protection.
Findings are stated directionally, as the paper reports them — we are not attaching invented effect sizes.
Blocking bots reduces traffic
Large publishers who block GenAI bots via robots.txt see reduced website traffic compared to comparable publishers who don't block. This is the core evidence behind the page's headline takeaway.
Publishers shift to richer content
Large publishers shift toward content that's richer and harder for LLMs to replicate, without increasing text volume.
Editorial hiring rises
The share of new editorial / content-production job postings rises over time at publishers responding to GenAI.
Strategic Response of News Publishers to Generative AI — Hangcheng Zhao (Rutgers Business School) & Ron Berman (The Wharton School, University of Pennsylvania), April 15, 2026 (staggered / synthetic difference-in-differences).GenAI can lower publisher demand (consumer substitution, reduced newsroom-employee demand, more AI “slop”) but can also act as a traffic-referral and discovery channel that increases demand — the strategic response a publisher chooses is what these findings measure.
The site inventory behind this page
usNewspaperPublisherInventory.xlsx is a real-site inventory, not a robots.txt classification — it grounds the ~7,000-site universe the prevalence audit above was run against.
- 6,963 unique included newspaper hosts
- 9,543 included title candidates (of 10,060 candidate records)
- Audit-tier coverage: Tier 1 (recent evidence) 1,519 · Tier 2 (corroborated) 3,391 · Tier 3 (directory-only) 2,053
- Benchmark: Medill counted 5,419 active local newspapers (Sept 2025) — this inventory is intentionally over-inclusive
An example from the anti-pattern cohort
Republic-Times is one of the real sites in our anti-pattern cohort — it runs the GPTBot-disallow-then-wildcard config described above.
A few more real US newspapers in the inventory, for scale — not individually classified for AI-blocking here:
Arizona Republic (arizonarepublic.com), Albany Times Union, The Durango Herald, Ocala Gazette, Fauquier Times, Worcester Herald.