To get a WordPress site cited by ChatGPT, Google AI Overviews, Perplexity, and the rest, you have three jobs: let the AI search crawlers reach your pages, write each page so a single passage can answer a question on its own, and then measure what happens. Most sites that never show up in AI answers are failing the first job without knowing it.
Last updated: November 1, 2026.
Disclosure: some links in this post are affiliate links, marked as sponsored. If you sign up through them I may earn a commission at no extra cost to you. I only recommend hosts I have used.
If you want the background on what AI optimization is and how it differs from SEO, start with What Is AIO? AI Optimization Explained. This post is the hands-on checklist.
Step 1: Know which AI crawlers matter
Each AI company runs more than one bot, and they do different jobs. The important split is between bots that power search and answers (block them and you disappear from that tool’s answers) and bots that collect training data (block them and you only opt out of future model training).
| User agent | Company | What it does | If you block it |
|---|---|---|---|
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode | You drop out of Google entirely. Never block this. | |
| Google-Extended | A robots.txt token for Gemini training and grounding in some other Google products | Opts out of that use. Does not affect Search or AI Overviews. | |
| Bingbot | Microsoft | Crawls for Bing, which also feeds Copilot | You lose Bing and Copilot visibility |
| OAI-SearchBot | OpenAI | Finds pages to show in ChatGPT search answers | You won’t appear in ChatGPT search answers |
| GPTBot | OpenAI | Collects content that may be used to train OpenAI models | Opts out of training only |
| ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user asks it to | OpenAI says robots.txt rules may not apply, since a person asked |
| Claude-SearchBot | Anthropic | Indexes pages for Claude’s search results | Less visibility in Claude’s search answers |
| ClaudeBot | Anthropic | Collects content that may be used to train Claude | Opts out of training only |
| Claude-User | Anthropic | Fetches a page when a Claude user asks a question | Claude can’t read your page for that user |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers | You won’t be surfaced in Perplexity search |
Sources: OpenAI crawler docs, Anthropic’s help center article on its crawlers, and Google’s AI features documentation, which is explicit that Googlebot rules are the control for AI Overviews.
Step 2: Check and fix your robots.txt
Open yourdomain.com/robots.txt in a browser. If you see Disallow: / under OAI-SearchBot, Claude-SearchBot, PerplexityBot, or User-agent: *, that’s your problem.
A common middle ground is to allow the search bots and opt out of training. It looks like this:
# Show up in AI search and AI answers
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Everyone else
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/wp-sitemap.xmlSwap in your own domain on the sitemap line. OpenAI says changes to robots.txt take about 24 hours to reach its systems, so don’t panic if nothing changes the same afternoon. Whether you block training bots at all is a judgment call. Some site owners figure being in the training data helps brand recall later. I don’t think there is a wrong answer, as long as you block on purpose and not by accident.
Where robots.txt lives on WordPress
Unless a physical robots.txt file sits in your site’s root folder, WordPress generates one on the fly. That’s how this blog works: I checked, and there is no file on disk. The generated version lists my sitemaps and blocks only /wp-admin/, so every AI crawler is welcome here. You have three ways to change a virtual robots.txt:
- Your SEO plugin’s robots.txt editor, if you run one (Yoast, Rank Math, and All in One SEO all have one).
- Upload a real
robots.txtfile to the root folder. It overrides the virtual one completely, so include everything you need. - Add rules with the
robots_txtfilter in a small must-use plugin. This is what I prefer because it survives theme changes and needs no plugin:
<?php
// wp-content/mu-plugins/ai-robots.php
// Adds AI crawler rules to the robots.txt WordPress generates.
add_filter( 'robots_txt', function ( $output, $public ) {
if ( ! $public ) {
return $output; // Site is set to discourage search engines.
}
$output .= "\nUser-agent: GPTBot\nDisallow: /\n";
$output .= "\nUser-agent: ClaudeBot\nDisallow: /\n";
$output .= "\nUser-agent: Google-Extended\nDisallow: /\n";
return $output;
}, 10, 2 );While you are in there, check Settings > Reading. If “Discourage search engines from indexing this site” is ticked, WordPress tells every crawler to stay away. It gets left on after launches more often than anyone admits.
Step 3: Check your firewall and CDN
robots.txt is a polite request. Firewalls are not. If your CDN or security plugin blocks a bot outright, your robots.txt rules don’t matter.
- Cloudflare. Starting July 1, 2025, Cloudflare began blocking known AI crawlers by default on newly added domains. If your site moved to Cloudflare after that, look at the AI crawler settings in your dashboard and decide which bots to let through. Cloudflare can control search bots and training bots separately.
- Security plugins. Some bot-blocking rules in security plugins lump AI crawlers in with scrapers. Check the blocked traffic log.
- Host-level protection. Many managed hosts run bot protection at the edge. Usually it leaves verified crawlers alone, but if your site disappears from AI answers after a move, ask support.
A quick test from the command line:
curl -s https://www.example.com/robots.txt
# Rough check for user-agent based blocking (a 403 here is a red flag)
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://www.example.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "PerplexityBot" https://www.example.com/This only catches blocks based on the user agent string. A firewall that verifies real bots by IP address may treat your test differently from the actual crawler, so treat a 200 as “probably fine” and a 403 as “definitely look into it.”
Step 4: Get indexed in Google and Bing
AI answers are built on search indexes, so the boring setup work still pays:
- Verify your site in Google Search Console and submit your sitemap. WordPress has a built-in one at
/wp-sitemap.xmlunless an SEO plugin or Jetpack replaced it (this blog uses Jetpack’s/sitemap.xml). - Verify in Bing Webmaster Tools too. People skip Bing, but Copilot is built on it, and Bing now reports how often you get cited in AI answers.
- Turn on IndexNow if your SEO plugin or host supports it. It pings Bing and other participating engines when you publish or update a post, instead of waiting for a recrawl.
Step 5: Make every page easy to quote
Once crawlers can reach you, the content has to be worth lifting. AI tools quote passages, so write in passages:
- Lead with the answer. The first two sentences under the title should answer the question the title asks. Same rule under each H2.
- Use question headings where it is natural, like “How long does a WordPress migration take?” That mirrors how people prompt AI tools.
- One idea per paragraph, and name the subject instead of saying “it.” A paragraph that starts “This plugin also…” can’t be quoted alone.
- Put comparisons in tables and steps in numbered lists.
- Date your content. A visible “Last updated” line, and real updates when facts change.
- Add an FAQ section for the questions people actually ask, with FAQPage structured data that matches the visible text word for word. Google is clear that structured data should match what’s on the page.
- Keep key content in the HTML. Many AI crawlers fetch the raw page and run little or no JavaScript, so text that only loads through scripts, sliders, or tabs may never be seen.
Here is the difference in practice. A typical old intro:
“In today’s fast-paced digital world, website speed matters more than ever. In this article, we’ll explore some of the many factors that can affect how quickly your WordPress site loads.”
The AIO version:
“The fastest way to speed up a slow WordPress site is to fix hosting and caching first, then images, then plugins. Most sites get the biggest gain from page caching, which a good managed host handles for you.”
The second one can be quoted as a complete answer. The first one says nothing.
Step 6: Measure AI traffic and citations
Measurement is patchy, but you can get a decent picture:
- GA4. In Explore, build a free-form report with Session source as the dimension and filter it with this regex:
chatgpt\.com|perplexity\.ai|gemini\.google\.com|copilot\.microsoft\.com|claude\.ai. ChatGPT usually addsutm_source=chatgpt.comto the links it shows, which helps. Two caveats: many AI clicks arrive with no referrer and land in Direct, and in October 2026 a ChatGPT bug stripped that tag for about a week before OpenAI fixed it. Assume your AI numbers are an undercount. - Bing Webmaster Tools AI Performance. Shows total citations in Copilot and Bing AI answers, which pages get cited, and the grounding queries behind them.
- Search Console. AI Overviews and AI Mode clicks are mixed into the regular Web numbers. You can’t separate them, but sudden drops in clicks with steady impressions are often an AI Overview taking the click.
- Manual checks. Keep a list of ten questions your site should answer. Ask them in ChatGPT, Perplexity, and Google once a month and write down who gets cited.
The short checklist
- robots.txt allows OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and Bingbot
- “Discourage search engines” is off
- CDN and security plugin aren’t blocking AI search bots
- Sitemap submitted to Google Search Console and Bing Webmaster Tools
- Every key page answers its question in the first two sentences
- Comparisons in tables, steps in lists, FAQ with matching structured data
- Visible “Last updated” dates
- AI referral report saved in GA4
Fast pages for readers and crawlers
A crawler that times out can’t cite you. This blog runs on Pressable, which puts Edge Cache in front of WordPress and includes staging sites, so I can test robots.txt and plugin changes before they go live.
FAQ
How do I get my website to show up in ChatGPT?
Allow OAI-SearchBot in your robots.txt and firewall, make sure your pages are indexed in search engines, and write pages that answer a specific question in the first two sentences. OAI-SearchBot is the crawler OpenAI uses to find pages for ChatGPT search answers.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot collects content for model training. Search visibility in ChatGPT is controlled by OAI-SearchBot, so you can block GPTBot and still appear in ChatGPT search answers.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google says Googlebot robots.txt rules control crawling for Search, including AI Overviews and AI Mode. Google-Extended only covers Gemini training and grounding in some other Google products.
Where is the robots.txt file in WordPress?
Unless a physical robots.txt file exists in the site root, WordPress generates a virtual one. You can change it with an SEO plugin, by uploading a real robots.txt file, or with the robots_txt filter in a must-use plugin.
How do I track traffic from ChatGPT and Perplexity in Google Analytics 4?
In GA4 Explore, use Session source as the dimension and filter with a regex such as chatgpt.com, perplexity.ai, gemini.google.com, copilot.microsoft.com, and claude.ai. Expect an undercount, because many AI clicks arrive without a referrer and show up as Direct.
Related posts
- What Is AIO? AI Optimization Explained (and How It Differs From SEO)
- How to Speed Up WordPress: Fixes That Start With Your Host
- The Almost Perfect .htaccess File for WordPress (2026 Update)
- Should You Use a Google Analytics Plugin for WordPress? (Updated for GA4)
- WP Engine and Cloudflare: Advanced Network, GES, and Your Own Account