Skip to content
WP Visibility

Everyday SEO

Set the AI Crawler Policy and Serve llms.txt

Two switches, one for model training crawlers and one for AI search crawlers, write the matching robots.txt rules. A third switch serves an llms.txt file with an editable body.

Updated September 22, 2026

Docs

Open WP Visibility → Settings and scroll to the AI Visibility section. Set Model training crawlers to Allow or Block, and AI search crawlers likewise. Blocking a group adds its crawlers to your robots.txt under a single Disallow: /. Blocking training costs no visibility in search or in AI answers; blocking AI search removes your site from the answers of the engines that respect the rule. Serve an llms.txt file answers /llms.txt with a generated page list and, if you write one, your own text above it.

Choose the policy

Setting Options Effect
Model training crawlers Allow, Block Block opts your content out of future model training runs for the listed crawlers.
AI search crawlers Allow, Block Block removes the site from the listed engines’ answers and citations. The screen warns when you choose it.
Per-bot overrides Class default, Allow, Block, per crawler Overrides the group setting for one crawler.

Resolution is per-bot override first, then the group setting; anything unset is allow. The module is on by default; its output is on the front end only.

These are published instructions, not access controls. A crawler operator decides whether to honor them, and blocking does not remove content already collected.

The crawlers in each group

Model training: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (the Gemini training opt-out; it does not affect Google Search), Applebot-Extended (Apple’s training opt-out; regular Applebot is unaffected), CCBot (Common Crawl, whose corpus feeds many training sets), meta-externalagent (Meta), and Bytespider (ByteDance).

AI search: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude search), PerplexityBot, ChatGPT-User and Claude-User (page fetches made on behalf of a user in a conversation), and DuckAssistBot (DuckDuckGo).

The list ships with the plugin and is updated in releases. Developers can add crawlers with the wpvis_ai_bots filter; an added crawler follows its group’s setting.

What the policy writes to robots.txt

WordPress builds robots.txt on request, and WP Visibility appends to it. With both groups blocked and no overrides, the addition is:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: DuckAssistBot
Disallow: /

With nothing blocked, nothing is added. The Content-Signals policy field publishes an optional Content-Signal: line as well, for example search=yes, ai-input=no, ai-train=no; leave it empty to publish none. The Sitemaps module adds the Sitemap: line separately. Check the result with the View robots.txt link in the section.

Two conditions make the file different from what you set. If a real robots.txt file exists in your site’s root folder, the web server serves that file and WordPress never runs, so nothing here is published until the file is removed. And while Settings → Reading discourages search engines, WordPress itself serves Disallow: / for every crawler.

Serve an llms.txt file

Turn on Serve an llms.txt file. The plugin answers https://your-site.com/llms.txt as plain text; no file is written and no rewrite rule is needed. It is off by default and marked experimental on the screen: measured use of these files by AI crawlers is close to zero, and publishing one does not establish that any assistant reads it. The document looks like this:

# Your site name

> Your tagline

Your own text, if you wrote any.

## Pages

- [Post title](https://your-site.com/post/): the first 120 characters of the excerpt

The Pages list holds up to 50 published posts and pages, most recently modified first. Noindexed and password-protected content is left out, and so is any post type ticked under Indexing → Noindex these post types. There is no per-post “leave me out of llms.txt” switch; Noindex is the way to drop one page from the list. Check the output with the View llms.txt link. The file is served with an X-Robots-Tag: noindex header so it does not appear in search results itself.

Edit the llms.txt body

Once the switch is on, an llms.txt body field appears. Whatever you write there, in Markdown, goes after the site name and tagline and before the generated page list, up to 64,000 bytes. Leave it empty to publish the generated document only. The heading and tagline always come from your site title and tagline; to change the whole document, use the wpvis_llms_txt filter in code.

A connected assistant can read and propose changes to this body through the review queue like any other supported change.

What this does not do

  • It does not detect or edit a physical robots.txt file.
  • It does not block crawlers at the server. For that, use your host’s or CDN’s bot rules.
  • It does not add per-page AI rules. The robots meta directives for snippet, image and video preview length live in the Indexing section and apply to search engines generally.