AI crawlers are bots that fetch your Shopify pages for AI engines: GPTBot and OAI-SearchBot (OpenAI/ChatGPT), ClaudeBot (Anthropic), PerplexityBot (Perplexity), and Google-Extended (Gemini). Some collect training data, others fetch live pages to answer a question. To be cited, allow them explicitly in robots.txt and serve clean, server-rendered HTML.
Before an AI engine can cite your Shopify store, a crawler has to fetch it. These bots are the gatekeepers of AI visibility — and unlike Googlebot, most merchants have never heard of them. This guide explains the AI crawlers that matter, what each one does, and how to make sure they are allowed to read your store.
What is an AI crawler?
An AI crawler is an automated bot that fetches web pages on behalf of an AI company. Like Googlebot, it requests your pages, reads the HTML, and stores what it finds — but the destination is an AI system rather than a traditional search index. Some crawlers gather data to train models; others fetch pages in real time to answer a specific user question with a citation.
Every crawler identifies itself with a user-agent string — like GPTBot or PerplexityBot — and you control access to each one through your robots.txt file. That makes robots.txt the single most important lever for AI visibility on Shopify: it decides who is even allowed to read your store.
The AI crawlers that matter for Shopify
You do not need to memorize every bot, but these are the ones worth knowing and, in most cases, allowing.
| Crawler | Operator | What it does | User-agent |
|---|---|---|---|
| GPTBot | OpenAI | Crawls pages to train and improve models | GPTBot |
| OAI-SearchBot | OpenAI | Fetches pages to surface in ChatGPT search results | OAI-SearchBot |
| ChatGPT-User | OpenAI | Fetches a page live when a user or action requests it | ChatGPT-User |
| ClaudeBot | Anthropic | Crawls pages to train Anthropic's Claude models | ClaudeBot |
| Claude-User | Anthropic | Fetches a page live when a Claude user's request needs it | Claude-User |
| Claude-SearchBot | Anthropic | Indexes pages for Claude's search answers | Claude-SearchBot |
| PerplexityBot | Perplexity | Indexes pages to cite in Perplexity answers | PerplexityBot |
| Google-Extended | Controls whether content is used by Gemini and Vertex AI | Google-Extended | |
| Applebot-Extended | Apple | Controls use of content for Apple AI features | Applebot-Extended |
Two kinds of bots: training vs live retrieval
The crawlers split into two functional groups, and the difference should shape how you treat each.
Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) collect content to train or ground models. Allowing them increases the chance your brand becomes part of what a model knows — useful for the from-memory answers engines give without citing a source.
Retrieval crawlers (OAI-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Claude-SearchBot) fetch pages live to answer a specific question, usually with a link. These are the ones most directly tied to citations and click-through traffic — you almost always want them allowed.
Blocking training crawlers only limits whether your content trains future models. Blocking retrieval crawlers like OAI-SearchBot and PerplexityBot directly costs you citations and traffic today. For most Shopify stores, allowing retrieval crawlers is a clear win.
How to allow AI crawlers on Shopify
Shopify serves robots.txt automatically, but you can customize it by adding a robots.txt.liquid template — the only method Shopify provides to add your own rules. Here is what the job entails.
A custom robots.txt template
The work happens in your theme code: a robots.txt.liquid template that controls what your store's robots.txt serves. There is no settings toggle for this — it is a code-level change to your live theme.
The pitfall: it replaces Shopify's defaults
A custom robots.txt.liquid replaces Shopify's default robots.txt output entirely. Done wrong, it can silently break the rules that already protect pages like checkout, cart and internal search — with no error or warning that anything changed. This is where most DIY attempts go wrong.
Explicit Allow rules per crawler
Each crawler you want reading your store — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended — needs its own explicit user-agent entry so there is zero ambiguity about access.
Verify and keep it current
Confirm the output at yourstore.com/robots.txt — and re-check it whenever the AI-crawler landscape shifts, which is often.
Each crawler's entry follows the same simple pattern:
User-agent: GPTBot
Allow: /Repeat the same pattern for OAI-SearchBot, PerplexityBot, Google-Extended and the others.
The bot list drifts. OpenAI alone runs three user-agents, Anthropic another three, and new engines keep launching crawlers of their own. A robots.txt written once and forgotten quietly goes stale — someone has to own keeping it current.
Should you allow or block them?
For most Shopify merchants the answer is: allow the crawlers you want to be cited by. Blocking is a deliberate choice, not a safe default.
If your goal is AI visibility, allow at least the retrieval crawlers — OAI-SearchBot, PerplexityBot and ChatGPT-User — because they are what put you into cited answers. Blocking them to keep content out of training also removes you from the live answers buyers see. Block selectively, not by default.
Both pitfalls above — overriding Shopify's defaults safely and keeping up with the drifting bot list — are exactly what the AI-Ready Kit Lite solves: a maintained, defaults-safe robots.txt that keeps Shopify's protective rules intact while welcoming every AI crawler that matters, plus a custom llms.txt and Organization schema. Managing several stores? The Agency kit ($599 launch price, white-label, 5 seats) rolls it out across clients. See all AI-Ready Kits.
Frequently asked questions
Where is the robots.txt file on Shopify, and can I edit it?
Shopify serves robots.txt automatically, but you can customize it by adding a templates/robots.txt.liquid file in your theme code. That is the only method Shopify provides to add Allow rules for AI crawlers.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls pages to train OpenAI models; OAI-SearchBot fetches pages to include in ChatGPT search results and citations. If you want to appear in ChatGPT answers with a link, OAI-SearchBot is the one to especially allow.
Will allowing AI crawlers slow down or overload my store?
For a typical Shopify store, no. These crawlers respect robots.txt and crawl at modest rates. The upside — being readable and citable — far outweighs the negligible extra load for most merchants.
What is llms.txt and do I need it?
llms.txt is an emerging plain-text file that gives AI engines a curated map of your most important pages and context. It is not required, but it complements robots.txt by helping crawlers find and understand your best content. A Shopify store can add one as a page or template.
Should I block AI crawlers to protect my content?
Only with a specific reason. Blocking training crawlers limits model training on your content; blocking retrieval crawlers costs you citations and traffic now. Most stores allow retrieval crawlers and decide on training crawlers case by case.
Get your robots.txt welcoming the AI crawlers above — without breaking Shopify's defaults and without babysitting the bot list yourself. The AI-Ready Kit Lite ships it maintained and defaults-safe alongside llms.txt and Organization schema, or book expert help.