The anti-pattern
A robots.txt that blocks GPTBot by name, then opens the door to everyone.
This is the config we see most often on sites in the anti-pattern cohort: a specific User-agent: GPTBot rule set to Disallow: /, immediately followed by a wide-open wildcard rule.
User-agent: GPTBot
Disallow: /
User-Agent: *
Visible to Google = visible to AI.
The wildcard rule (User-agent: *) covers every crawler that isn’t named above it, including Googlebot. Naming one AI bot and leaving the wildcard open doesn’t make the site invisible to AI; it just blocks one named crawler while every other path in (including AI systems that read what Google indexes) stays open. It likely isn’t achieving the intended goal.
Worse: per the paper below, large publishers that block this way see reduced website traffic compared to publishers that don’t. The block correlates with a cost, not a protection.
AI anti-pattern prevalence
Roughly 3 in 10 newspaper sites run the detrimental config.
From our robots.txt audit of ≈7,000 US newspaper sites (6,963 unique hosts in our inventory of US newspaper sites). This split is our own separate robots.txt audit. The inventory itself carries no per-site AI-block classification.
Sample: 6,963 unique US newspaper websites from our inventory of newspaper sites. The inventory only tells us which sites exist; the 30/55/15 split above comes from a separate check of each site's robots.txt file, not from the inventory.
What the paper found
Blocking GenAI bots correlates with less traffic, not more protection.
Findings are stated directionally, as the paper reports them. We are not attaching invented effect sizes.
Blocking bots reduces traffic
Large publishers who block GenAI bots via robots.txt see reduced website traffic compared to comparable publishers who don't block. This is the core evidence behind the page's headline takeaway.
Publishers shift to richer content
Large publishers shift toward content that's richer and harder for LLMs to replicate, without increasing text volume.
Editorial hiring rises
The share of new editorial / content-production job postings rises over time at publishers responding to GenAI.
Strategic Response of News Publishers to Generative AI. Hangcheng Zhao (Rutgers Business School) & Ron Berman (The Wharton School, University of Pennsylvania), April 15, 2026 (staggered / synthetic difference-in-differences). GenAI can lower publisher demand (consumer substitution, reduced newsroom-employee demand, more AI “slop”) but can also act as a traffic-referral and discovery channel that increases demand. The strategic response a publisher chooses is what these findings measure.
The site inventory behind this page
Our inventory of US newspaper sites is a real-site list, not a robots.txt classification. It grounds the ~7,000-site universe the prevalence audit above was run against.
- 6,963 unique newspaper websites
- 9,543 newspaper titles included (out of 10,060 titles we found)
- Evidence strength per website: 1,519 backed by recent evidence · 3,391 corroborated by other sources · 2,053 listed in a directory only
- Benchmark: Medill counted 5,419 active local newspapers (Sept 2025). This inventory is intentionally over-inclusive
An example from the anti-pattern cohort
One newspaper site in our audit runs the GPTBot-disallow-then-wildcard config described above.

