AI Crawlers Turned Away More Than Twice as Often, a Check to Run Before an AI Visibility Audit

• Search and user-triggered requests grew from 6.51% to 11.92% of AI crawl activity between the second and third quarters of 2026.
• A site can allow AI search crawlers in its robots.txt file and still block them through server, firewall or CDN settings.
• Zen Media's AI visibility audits measure whether brands appear in AI answers across Claude, ChatGPT, Gemini, Perplexity and Grok.

Miami, FL, Oct. 09, 2026 (GLOBE NEWSWIRE) -- Zen Media, which specializes in AI visibility for brands, today issued guidance on how to do an AI visibility audit for companies that rank on Google but go unnamed in AI answers. The guidance starts with AI crawlers and follows preliminary Cloudflare Radar data, analyzed by TechnologyChecker.io, showing that websites on Cloudflare's network returned an access-denied response to 13.47% of AI bot requests in the third quarter of 2026, up from 5.98% in the same quarter a year earlier. A page that AI crawlers cannot reach cannot serve as a source when an AI platform pulls current web information.

"Sometimes the fix is one line in a robots.txt file," said Sarah Evans, CEO of Zen Media. "Sometimes it's a firewall setting nobody on the marketing team knew existed."

What the Crawler Data Shows

A larger share of AI crawling now supports search and user-triggered requests. By declared purpose, those requests accounted for 11.92% of AI crawl activity in the third quarter of 2026, compared with 6.51% in the second quarter.

Sites are also treating training bots and answer bots differently. In a Sept. 28 snapshot of 4,242 robots.txt files, OpenAI's GPTBot, which collects content that may be used for model training, was blocked 1.60 times for every time it was allowed. OAI-SearchBot, which surfaces pages in ChatGPT search, was blocked 0.91 times for every allow.

The refused requests point to a different layer. A 403 response is an HTTP access denial rather than a robots.txt directive, so a site can welcome an AI search crawler in robots.txt and still turn it away through firewall, CDN or bot-protection rules. A complete access check covers both: the robots.txt file and the firewall, CDN and server rules that apply to the user agents major AI platforms publish. Once a crawler reaches a page, the information still needs to be clearly stated and easy to extract, such as direct answers and specifications within the relevant section.

What an AI Visibility Audit Measures

An AI visibility audit picks up where a crawler check stops, measuring whether AI assistants name the brand when buyers prompt them about the category. Zen Media's audit measures where the brand appears across AI answers and how that visibility compares with competitors.

The audit runs each buyer prompt through the ZAVI™ AI visibility engine on Claude, ChatGPT, Gemini, Perplexity and Grok. Each run produces a visibility score for every engine and a competitor share-of-voice view, and shows how often the brand appears as the first named recommendation, among the first three brands named or among the first five brands named. Zen Media reports this starting point as Answer Share™, before any new content goes live.

Building the Third-Party Sources AI Systems Cite

AI answers also draw on sources outside a company's own website. Zen Media's Published Monthly™ program maps the prompts where a brand is missing and places coverage in media sources that can surface in AI answers, linking each placement back to the brand's owned content. After publication, AI citation reports track changes in AI answers, Google AI Overviews and keyword rankings against the brand's starting point.

Companies can book an AI visibility audit through Zen Media's AI Accelerator or Published Monthly™ programs.

Frequently Asked Questions

Question: How is an AI visibility audit different from an SEO audit?
Answer: An SEO audit checks how a site ranks in search results and how search engines crawl and index its pages. An AI visibility audit goes a step further and measures whether AI assistants such as ChatGPT, Gemini and Perplexity name the brand when buyers ask about its category. A site can score well on an SEO audit and still go unnamed in AI answers.

Question: What is the difference between GPTBot and OAI-SearchBot?
Answer: GPTBot crawls content that may be used to train OpenAI's models. OAI-SearchBot surfaces websites in ChatGPT search. OpenAI controls them independently, so a site can block training while still allowing its pages to appear in ChatGPT search.

Question: Does blocking AI crawlers remove a brand from ChatGPT?
Answer: Usually not completely. ChatGPT can still mention a brand from existing model knowledge or third-party sources. Sites that block OAI-SearchBot will not appear in ChatGPT search answers, according to OpenAI, though they can still appear as navigational links.

Question: Do Google AI Overviews and ChatGPT use the same sources?
Answer: Google AI Overviews and ChatGPT choose sources through different systems, so a page or brand surfaced by Google is not guaranteed to appear in ChatGPT for the same prompt.

Question: How long does it take for changes to show up in AI answers?
Answer: It depends on how the assistant produces the answer. Search-based answers can reflect updated pages after those pages are recrawled or retrieved again. Information from the model's existing learned knowledge may take longer to change and does not update simply because a webpage was edited.

Sources
Cloudflare Radar data as analyzed in TechnologyChecker.io's "robots.txt AI Crawler Report Q3 2026: Who Gets Blocked," updated Oct. 1, 2026. September and third-quarter 2026 figures are preliminary. Crawler descriptions from OpenAI's published crawler documentation.

About Zen Media

Zen Media is a strategic communications and marketing company helping brands shape how they are discovered, understood, and chosen across media, search, and AI. The company combines public relations, content, demand generation, and proprietary AI visibility technology to build authority across the buyer journey, from earned media and search to AI-generated answers. Zen Media works with companies across industries and stages, from growth-stage brands to global enterprises, that want to build authority, demand, and visibility across modern discovery channels.

Beyza Aydogan
Zen Media
pr@zenmedia.com