Set the AI Crawler Policy and Serve llms.txt

Open WP Visibility → Settings and scroll to the AI Visibility section. Set Model training crawlers to Allow or Block, and AI search crawlers likewise, then click Save changes. Blocking a group adds its crawlers to your robots.txt under a single Disallow: /. Blocking training does not affect Google Search or ChatGPT search, according to Google and OpenAI (linked below); blocking AI search can remove your site from the answers of the engines that respect the rule. Serve an llms.txt file answers /llms.txt with a generated page list and, if you write one, your own text above it.

Choose the policy

Resolution is per-bot override first, then the group setting; anything unset is allow. The AI Visibility module is on by default; its output is on the front end only.

These are published instructions, not access controls. A crawler operator decides whether to honor them, and blocking does not remove content already collected.

The crawlers in each group

Model training: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google’s token for Gemini training and for grounding in Gemini Apps; it does not affect Google Search), Applebot-Extended (Apple’s training opt-out; pages that block it can still appear in Apple’s search results), CCBot (Common Crawl, whose data has been a primary training source for many large language models), meta-externalagent (Meta, used for AI model training and for indexing content to improve its products), and Bytespider (we found no operator documentation for it).

AI search: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude search), PerplexityBot (Perplexity), ChatGPT-User and Claude-User (page fetches made on behalf of a user in a conversation; OpenAI says robots.txt rules may not apply to ChatGPT-User), and DuckAssistBot (DuckDuckGo). Operator pages checked September 27, 2026.

The built-in list ships with the plugin and changes only with plugin releases. Developers can add crawlers with the wpvis_ai_bots filter; an added crawler follows its group’s setting.

What the policy writes to robots.txt

WordPress builds robots.txt on request, and WP Visibility appends to it. With both groups blocked and no overrides, the addition is:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: DuckAssistBot
Disallow: /

With nothing blocked, no crawler lines are added. The Content-Signals policy field publishes an optional Content-Signal: line as well, for example search=yes, ai-input=no, ai-train=no; leave it empty to publish none. The Sitemaps module adds the Sitemap: line separately. Check the result with the View robots.txt link in the section.

Two common conditions make the file different from what you set. If a real robots.txt file exists in your site’s root folder, the web server serves that file instead of passing the request to WordPress (the standard WordPress rewrite rules skip WordPress for any path that is an existing file, checked September 27, 2026), so nothing here is published until the file is removed. And while Settings → Reading discourages search engines, WP Visibility 2.10.3 adds nothing to robots.txt, so the crawler rules above are not published there. WordPress itself no longer adds a blanket Disallow: / to robots.txt on such a site; since version 5.3 it relies on a noindex robots meta tag instead, per the do_robots() and wp_robots_no_robots() references, checked September 27, 2026.

Serve an llms.txt file

Turn on Serve an llms.txt file. The plugin answers https://your-site.com/llms.txt as plain text; no file is written and no rewrite rule is needed. It is off by default and marked experimental on the screen, and publishing one does not establish that any assistant reads it. The document looks like this:

# Your site name

> Your tagline

Your own text, if you wrote any.

## Pages

- [Post title](https://your-site.com/post/): up to about 120 characters of the excerpt, or of the content when there is no excerpt

The Pages list holds up to 50 published posts and pages, most recently modified first. Noindexed and password-protected content is left out, and so is any post type ticked under Indexing → Noindex these post types. There is no per-post “leave me out of llms.txt” switch; Noindex is the way to drop one page from the list. After saving the switch and reloading Settings, check the output with the View llms.txt link. The file is served with an X-Robots-Tag: noindex header, which asks search engines not to index the file itself.

Edit the llms.txt body

Once the switch is saved on and you reload Settings, an llms.txt body field appears. Whatever you write there, in Markdown, goes after the site name and tagline and before the generated page list, up to 64,000 bytes; anything longer is cut off. Leave it empty to publish the generated document only. The heading and tagline always come from your site title and tagline; to change the whole document, use the wpvis_llms_txt filter in code.

A connected assistant can read this body with the get-llms-txt ability and change it with update-llms-txt. When the Autopilot module is on and Assistant changes is set to Hold for my review, the change waits in the review queue like any other supported change; otherwise it is written straight away.

What this does not do

• It does not detect or edit a physical robots.txt file.
• It does not block crawlers at the server. For that, use your host’s or CDN’s bot rules.
• It does not add per-page AI rules. The robots meta directives for snippet length, image preview size and video preview length live in the Indexing section and apply to search engines generally.