← All articlesAI search
llms.txt and AI crawlers: what website owners should know
GPTBot, ClaudeBot, PerplexityBot and Google-Extended: how AI crawlers work, whether to allow them, and what llms.txt is.
AI companies use crawlers to read the web. Whether they can read your site affects whether AI assistants can learn about and cite you.
Common AI crawlers
- GPTBot and OAI-SearchBot (OpenAI)
- ClaudeBot (Anthropic)
- PerplexityBot (Perplexity)
- Google-Extended (controls use of content for Google's AI models)
Allow or block?
If you want AI assistants to recommend your business, you generally want search-related AI crawlers to access your public pages. Block only private or sensitive areas in robots.txt.
What is llms.txt?
llms.txt is a proposed plain-text file that summarises your site for AI systems. Support varies, and Google says it is not required for its AI features, but it is quick to add and harmless.
Focus on the basics
Fast pages, clear answers, schema markup and consistent business details matter more than any single file.
Have an idea or a business ready to grow?
