So, How Do You Make a Website AI-Crawlable?
Short version? Open the door to the right bots, then give them actual words to read. First, OAI-SearchBot, PerplexityBot and Googlebot have to make it past your robots.txt, and past your firewall too, which people forget. The words that matter should be in the HTML from the start, because text that JavaScript drops in later is a gamble. After that it’s housekeeping: clean 200 responses, and structured data that doesn’t contradict the page. None of it guarantees a mention. It just stops you being invisible by accident.
Ever typed your own business into ChatGPT and watched it rattle off a competitor’s services while you get a passing mention, if that? Your content may well be better. Often the culprit is dull. Maybe a security plugin is bouncing bots it hasn’t heard of. Maybe the service pages pull their text in with JavaScript, after the crawler has already taken its copy and gone.
Hence this guide, which is all plumbing. Content, reputation and how AI tools pick their sources get the full treatment in The Complete Guide to AI Search Visibility. What’s left is the technical side, and most of it you can check in an afternoon.
Crawlable vs Indexable vs Visible in AI Answers
People use these interchangeably, which muddies what you can fix. Crawlable is the low bar: a bot reached the page and could read it. Indexable asks more, because the platform needs permission to store the page and then has to decide it’s worth storing. And visible? That’s your business actually named in an AI answer, which depends on everything in this post plus plenty you don’t control.
So crawlability gets you to the starting line. Get it wrong and you’re out of the running before the quality of your content even comes into it.
Six Technical Checks That Help Every AI Crawler
1. Put Important Content in the HTML
Google renders JavaScript. OpenAI and Perplexity don’t say in their crawler documentation whether their bots do, so the safe assumption is that some won’t. Right-click a key page, choose “View page source”, and search for a sentence from your main content. If it isn’t there, it’s being added later by a script. Server-side rendering or static HTML fixes that on most modern sites. While the source is open, scroll for the H1. There should be just the one, with H2s following in a sensible order, and any heading saved as an image needs replacing with real text.
2. Write robots.txt Rules Crawler by Crawler
Each platform runs separate bots for separate jobs, and robots.txt lets you treat them differently. Plenty of businesses blocked “AI bots” wholesale a couple of years ago and never revisited it. Look at yoursite.co.uk/robots.txt and read it line by line. A blanket rule blocking all bots, or an old line aimed at OpenAI, may be keeping you out of AI search without anyone realising.
Example robots.txt for AI Search Crawlers
| # Allow AI search crawlers to find and cite the site User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / # Optional: opt out of OpenAI model training User-agent: GPTBot Disallow: / |
Adapt it to your site before pasting anything in. Blocking GPTBot is optional, a choice about training data, and whatever already protects your admin or checkout pages stays. Give it about a day; OpenAI and Perplexity both say updates aren’t picked up instantly.
3. Make Sure Your Server and Firewall Let Them In
Robots.txt is only half of it. Hand it a 403, a 5xx error or a “prove you’re human” page and it leaves with nothing. In our experience this is where most silent failures happen, because firewalls and CDNs rarely announce what they block. Several now include a switch for blocking AI crawlers, and depending on who you’re with and which plan you’re on, it may already be flipped. Have a look through your security plugin or CDN dashboard for anything mentioning bots or AI, and make sure the crawlers named here aren’t on a block list. Your developer, or the host’s support desk, can search the server logs for those user agents too. If they’ve never appeared, that tells you something. Handily, OpenAI and Perplexity publish the IP addresses their bots use, so a firewall rule can let the genuine article through while fakes using the same name stay out.
4. Keep Your Schema Honest
We get asked about “AI schema” a lot. There isn’t any, and Google says its AI features need nothing special. What’s worth having on a small business site is fairly standard: Organization or LocalBusiness on the homepage, Service on each service page, FAQPage where real FAQs appear. Google’s one firm rule is that the markup matches what visitors can see. So if your schema says you open on Saturdays and the page never mentions it, take that bit out.
5. Link Your Important Pages Together
Crawlers move around by following links, much like a visitor would. Pages listed only in the sitemap, or tucked four clicks deep, tend to get less attention. The fix is simple enough. Put each core service in the main menu, mention it in a couple of related blog posts, and write link text that would make sense on its own, such as “physiotherapy for sports injuries”. “Click here” tells a crawler nothing.
6. Spell Out the Basics in Plain Text
AI tools are often asked factual questions about a business. Where are you, what do you charge, do you cover Lurgan? Bury the answer in a PDF brochure or an image in the footer and there’s a decent chance it gets missed. Plain text works better. Your address, the towns you cover, phone and email, and a line on each core service should all sit on normal pages somewhere, worded the same way wherever they crop up.
Rather have someone else look under the bonnet? Our AEO and GEO team checks robots.txt, firewall rules, rendering and schema against each platform’s published documentation. Grab a free consultation slot and we’ll talk you through it, warts and all.
Platform-by-Platform Guidance
Quick caveat. These companies rename bots and tweak their rules now and again, and everything below comes from their own documentation as it stood in October 2026. Double-check the source pages before making any big change.
ChatGPT (OpenAI)
OpenAI separates search from training, which is the detail most people miss. OAI-SearchBot is the crawler that surfaces sites in ChatGPT search answers, and OpenAI says sites that block it won’t appear in those answers (though they can still show up as plain navigational links). GPTBot collects content that may be used to train OpenAI’s models. Blocking it has no stated effect on search. Then there’s ChatGPT-User. It only turns up when someone’s conversation needs a page fetched and doesn’t crawl on its own, and since a person kicked off the visit, OpenAI says robots.txt rules may not apply. If ChatGPT visibility matters to you, OAI-SearchBot is the one to allow.
Google Gemini
Here’s where Google differs. There’s no separate Gemini crawler. Google uses the user agents it already runs, and Google-Extended is simply a token you write into robots.txt. Disallow it and you’re telling Google not to use your content to train future Gemini models, or to ground answers in the Gemini app and on Vertex AI. Your presence in Google Search is unaffected, as Google is explicit that the token plays no part in whether you appear there or how you rank.
Which leaves you with a judgement call. Opting out of training through Google-Extended opts you out of grounding as well, and grounding is how the Gemini app pulls web content into its answers. AI Overviews and AI Mode in Google Search work differently. They draw on pages that are indexed and eligible for a snippet, with no extra technical requirements, and the controls are the usual ones: robots.txt for Googlebot, noindex, and snippet controls like nosnippet. URL Inspection in Search Console shows you exactly what Googlebot sees.
Perplexity
Perplexity’s set-up looks a lot like OpenAI’s. PerplexityBot surfaces and links websites in Perplexity’s search results, follows robots.txt, and according to Perplexity isn’t used to crawl content for AI foundation models. Perplexity-User visits pages in response to a user’s question and generally ignores robots.txt, since the user asked for it. Perplexity publishes IP lists for both, along with guidance on allowing them in Cloudflare and AWS WAF, matching on user agent and IP together.
AI Crawler Comparison Table
| Platform | Search crawler to allow | User-triggered fetcher | Training or grounding control |
|---|---|---|---|
| ChatGPT | OAI-SearchBot | ChatGPT-User | GPTBot (training) |
| Gemini | Googlebot (Google’s existing crawlers) | Not covered here | Google-Extended token (training and Gemini grounding) |
| Perplexity | PerplexityBot | Perplexity-User | PerplexityBot is not used for foundation model training |
A Quick AI Crawlability Checklist
- robots.txt allows OAI-SearchBot, PerplexityBot and Googlebot on the pages you want found.
- Firewall or CDN bot settings aren’t blocking those crawlers.
- Key pages return 200 responses, with no login walls or CAPTCHAs.
- Main content appears in “View page source”, not only after scripts run.
- Each page has one clear H1 and logical subheadings.
- Organization or LocalBusiness and Service schema match the visible text.
- Core service pages are linked from the navigation and related posts.
- Address, contact details and service areas appear as plain text and match everywhere.
Frequently Asked Questions
What is website AI crawlability?
In plain terms: can the bots behind ChatGPT, Gemini and Perplexity reach your pages, and once there, can they read them? Robots.txt, firewall rules, server responses and whether text sits in the HTML or arrives via scripts all feed into that.
Which crawlers do ChatGPT, Gemini and Perplexity use?
Each company does it a bit differently. OpenAI’s search crawler is OAI-SearchBot; Perplexity’s is PerplexityBot. Gemini borrows Google’s normal crawling, with a robots.txt token called Google-Extended deciding whether your content can be used for training and grounding. OpenAI and Perplexity also run bots that fetch pages on a user’s behalf (ChatGPT-User and Perplexity-User), and OpenAI has GPTBot for training.
If I block GPTBot, will my website disappear from ChatGPT?
No. OpenAI keeps these separate. GPTBot gathers training data, and ChatGPT search relies on OAI-SearchBot, so blocking the first leaves the second untouched.
Do I need an llms.txt file for AI search?
Not as far as these three are concerned. llms.txt doesn’t appear as a requirement in any of their crawler documentation, and Google says its AI features don’t need special text files. Time spent on clean HTML will go further.
Will fixing crawlability get my business cited by AI tools?
Nobody can promise you that, us included. Fixing crawlability puts your pages in front of these systems. What they then cite comes down to relevance, quality and the competition on that topic.
Where to Start This Week
If you only do one thing, open your robots.txt and your firewall’s bot settings side by side and compare them with the crawler table above. Ten minutes, maybe less. Next, pull up the source code of your three busiest pages and search for a sentence you know is on each one. If it’s missing, you’ve found your next job.
Will any of this get you cited? Possibly, but nobody can honestly guarantee it. Think of it as making sure your best pages are at least in the running. If you’d like a second pair of eyes, Creative Sweet’s AEO and GEO services start with exactly this kind of technical check. Book a free consultation and we’ll show you what each platform can see.
