Back to Blog
    TechnicalGEOAEO

    Complete llms.txt and robots.txt Setup Guide for AI Search 2025

    Step-by-step guide to configuring llms.txt and robots.txt for AI search visibility. Includes templates, examples, and best practices for making your site accessible to AI crawlers and agents.

    Julia Maehler··9 min read

    Configuring your site for AI visibility starts with two key files: robots.txt and llms.txt. This guide provides everything you need to set up both correctly.

    Understanding the Two Files

    robots.txt

    What it does: Controls which crawlers (including AI crawlers) can access which parts of your site.

    Location: yourdomain.com/robots.txt

    Who reads it: All search engine crawlers, AI crawlers, and well-behaved bots

    Impact: Blocking AI crawlers here prevents them from indexing your content entirely

    llms.txt

    What it does: Provides AI systems with structured context about your site, services, and content.

    Location: yourdomain.com/llms.txt (and optionally llms-full.txt)

    Who reads it: Currently limited adoption; designed for LLMs to understand your site better

    Impact: Helps AI systems understand your site's purpose and structure

    Part 1: robots.txt Configuration

    Basic Structure

    A robots.txt file consists of one or more rule sets, each targeting specific user agents:

    User-agent: [crawler name] Allow: [path] Disallow: [path]

    AI Crawlers to Know

    CrawlerCompanyPurpose
    GPTBotOpenAIPowers ChatGPT search and training
    OAI-SearchBotOpenAISpecifically for search features
    ChatGPT-UserOpenAIChatGPT browsing with user context
    ClaudeBotAnthropicPowers Claude's knowledge
    PerplexityBotPerplexityPowers Perplexity search
    Google-ExtendedGoogleGoogle AI features (Bard, AI Overviews)
    BytespiderByteDanceAI training and features
    CCBotCommon CrawlOpen dataset used for AI training
    anthropic-aiAnthropicClaude training data

    Recommended Configuration: Allow All AI Crawlers

    For maximum AI visibility, allow all AI crawlers:

    # Allow all standard crawlers User-agent: * Allow: /

    # Explicitly allow AI crawlers User-agent: GPTBot Allow: /

    User-agent: OAI-SearchBot Allow: /

    User-agent: ChatGPT-User Allow: /

    User-agent: ClaudeBot Allow: /

    User-agent: PerplexityBot Allow: /

    User-agent: Google-Extended Allow: /

    User-agent: Bytespider Allow: /

    User-agent: CCBot Allow: /

    User-agent: anthropic-ai Allow: /

    # Block sensitive areas from all crawlers User-agent: * Disallow: /admin/ Disallow: /private/ Disallow: /api/internal/

    # Sitemap location Sitemap: https://yourdomain.com/sitemap.xml

    Selective AI Crawler Access

    If you want some AI crawlers but not others (e.g., allow search but not training):

    # Allow search-focused crawlers User-agent: OAI-SearchBot Allow: /

    User-agent: PerplexityBot Allow: /

    User-agent: Google-Extended Allow: /

    # Block training-focused crawlers User-agent: GPTBot Disallow: /

    User-agent: CCBot Disallow: /

    User-agent: Bytespider Disallow: /

    Block All AI Crawlers

    If you don't want AI systems accessing your content:

    # Block all known AI crawlers User-agent: GPTBot Disallow: /

    User-agent: OAI-SearchBot Disallow: /

    User-agent: ChatGPT-User Disallow: /

    User-agent: ClaudeBot Disallow: /

    User-agent: PerplexityBot Disallow: /

    User-agent: Google-Extended Disallow: /

    User-agent: Bytespider Disallow: /

    User-agent: CCBot Disallow: /

    User-agent: anthropic-ai Disallow: /

    Warning: Blocking AI crawlers prevents your content from appearing in AI-generated responses. Consider whether the tradeoff is worth it for your business.

    Common Patterns

    E-commerce site:

    User-agent: * Allow: /

    # Block checkout and account pages Disallow: /checkout/ Disallow: /cart/ Disallow: /account/ Disallow: /admin/

    # Allow product and category pages Allow: /products/ Allow: /categories/ Allow: /blog/

    Sitemap: https://store.com/sitemap.xml

    SaaS application:

    User-agent: * Allow: /

    # Block application routes Disallow: /app/ Disallow: /dashboard/ Disallow: /api/

    # Allow marketing and documentation Allow: / Allow: /docs/ Allow: /blog/ Allow: /pricing/

    Sitemap: https://saas.com/sitemap.xml

    Publishing site:

    User-agent: * Allow: /

    # Allow everything except admin Disallow: /admin/ Disallow: /wp-admin/

    # Explicitly welcome AI crawlers User-agent: GPTBot Allow: /

    User-agent: ClaudeBot Allow: /

    User-agent: PerplexityBot Allow: /

    Sitemap: https://publisher.com/sitemap.xml

    Part 2: llms.txt Configuration

    The llms.txt Standard

    Proposed by Jeremy Howard of Answer.AI in September 2024, llms.txt provides a markdown-formatted file that helps LLMs understand your website.

    Basic llms.txt Structure

    # Your Site Name

    A one-sentence description of what your site offers and who it's for.

    Overview

    A paragraph explaining your organization, your expertise, and what makes your content authoritative.

    Key Resources

    About

    Information about the organization, team, or author credentials that establish authority.

    Contact

    How to reach you for inquiries: email, form link, etc.

    llms.txt Examples

    E-commerce Store:

    # TechGear Shop

    Premium electronics and accessories with expert reviews and buying guides.

    Overview

    TechGear Shop has been helping customers find the right technology products since 2015. Our team includes certified electronics experts who test every product we sell. We specialize in laptops, smartphones, audio equipment, and smart home devices.

    Key Resources

    • Product Catalog: Browse our full selection with detailed specs
    • Buying Guides: Expert advice for choosing the right products
    • Reviews: In-depth testing and comparisons
    • Deals: Current promotions and discounts
    • Support: Product help and warranty information

    Expertise

    Our review team has over 50 years combined experience in consumer electronics. All products undergo minimum 2-week testing before review publication.

    Contact

    support@techgearshop.com for product questions press@techgearshop.com for media inquiries

    SaaS Documentation:

    # DataFlow Platform

    Enterprise data integration platform connecting 200+ data sources.

    Overview

    DataFlow enables businesses to build data pipelines without code. Used by Fortune 500 companies for ETL, data synchronization, and analytics preparation.

    Documentation

    Resources

    • Blog: Product updates and data engineering best practices
    • Case Studies: How companies use DataFlow
    • Changelog: Recent updates and new features

    Support

    docs@dataflow.io for documentation feedback support@dataflow.io for technical issues

    Professional Services:

    # Martinez Legal Group

    Business law firm specializing in startup formation and venture capital transactions.

    Overview

    Martinez Legal Group provides legal services to technology startups and venture capital firms in the San Francisco Bay Area. Founded in 2010, we've helped over 500 startups from formation through exit.

    Practice Areas

    Resources

    • Startup Legal Guide: Free comprehensive guide for founders
    • Blog: Legal updates affecting startups
    • FAQ: Common legal questions answered

    Credentials

    Partners have represented companies acquired by Google, Meta, and Microsoft. Named "Top Startup Law Firm" by Silicon Valley Business Journal 2023-2025.

    Contact

    info@martinezlegal.com (415) 555-0100

    llms-full.txt: The Comprehensive Version

    The llms.txt specification also defines llms-full.txt—a single file containing all your important content, eliminating the need for AI to follow links.

    When to use llms-full.txt:

    • Documentation sites where content should be consumed together
    • Reference materials that benefit from complete context
    • Sites where you want to ensure AI has all information

    Structure:

    # Site Name

    Summary description

    Section 1: [Topic]

    Full content of the section, written in markdown.

    All the detailed information that would normally require following links.

    Section 2: [Topic]

    More complete content.

    Subsections as needed with full detail.

    Section 3: [Topic]

    Continue with all relevant content...

    Example snippet:

    # TechGear Reviews

    Expert electronics reviews and buying guides since 2015.

    Laptop Buying Guide 2025

    When choosing a laptop in 2025, consider these key factors:

    Processors

    Intel Core Ultra and AMD Ryzen 9000 series dominate the market. For most users:

    • Everyday use: Intel Core Ultra 5 or AMD Ryzen 5 provides excellent performance
    • Professional work: Intel Core Ultra 7 or AMD Ryzen 7 handles demanding applications
    • Creative/Gaming: Intel Core Ultra 9 or AMD Ryzen 9 for maximum performance

    Memory

    Minimum 16GB RAM for 2025. Consider 32GB if you: - Edit video or large images - Run virtual machines - Keep many browser tabs open - Use memory-intensive development tools

    Storage

    NVMe SSDs are standard. Minimum 512GB recommended, 1TB preferred.

    [Content continues with full buying guide...]

    Smartphone Buying Guide 2025

    [Full guide content...]

    Implementation Tips

    Keep it updated: llms.txt should reflect your current site structure. Update when you: - Add major new sections - Change site organization - Update key offerings

    Use relative URLs: Links should be relative paths (/docs/api) not absolute URLs, making the file portable.

    Write for machines and humans: llms.txt may be read by both LLMs and human developers. Keep it clear and well-organized.

    Don't duplicate robots.txt: llms.txt describes what your site is; robots.txt controls access. They serve different purposes.

    Part 3: Testing Your Configuration

    Testing robots.txt

    Google's robots.txt Tester: 1. Go to Google Search Console 2. Navigate to Settings > robots.txt 3. Test specific URLs against your rules

    Manual Testing: Visit yourdomain.com/robots.txt directly and verify the content.

    Common Issues:

    • File not at root (must be exactly /robots.txt)
    • Syntax errors (check for typos)
    • Rules in wrong order (more specific rules should come first)
    • Missing sitemap reference

    Testing llms.txt

    Manual Verification: Visit yourdomain.com/llms.txt and verify: - File loads correctly - Markdown renders properly - Links are valid - Content is current

    Validation Checklist:

    • H1 with site name present
    • Blockquote summary included
    • Key resources listed with working links
    • Contact information provided
    • Content is accurate and current

    Part 4: Monitoring AI Crawler Activity

    Server Log Analysis

    Check server logs for AI bot activity:

    # Search for AI crawlers in Apache/Nginx logs grep -E "GPTBot|ClaudeBot|PerplexityBot|Google-Extended" access.log

    Key Metrics to Track

    Crawl frequency: How often do AI bots visit? Pages crawled: Which content do they access? Response codes: Are they getting 200s or errors? Crawl depth: How deep into your site do they go?

    What Normal AI Crawler Behavior Looks Like

    • Respects robots.txt directives
    • Identifies via User-Agent string
    • Reasonable crawl rate (not hammering your server)
    • Accesses publicly available pages
    • Responds to crawl-delay directives

    The Reality Check

    llms.txt Adoption Status

    As of late 2025, llms.txt adoption is growing but impact is uncertain:

    Adoption numbers: - 844,000+ sites implementing llms.txt (per BuiltWith) - Major docs platforms (Mintlify) auto-generating llms.txt - Notable adopters: Anthropic, Cursor, Cloudflare

    The sobering reality: Research shows zero confirmed visits from major AI crawlers (GPTBot, ClaudeBot, PerplexityBot) to llms.txt files. No correlation found between having llms.txt and receiving AI citations.

    Recommendation: Implement llms.txt because: - It's low effort (minutes to create) - It may gain traction as the standard matures - It's good documentation regardless of AI impact - It doesn't hurt and might help

    But don't expect immediate impact on AI visibility. Focus your main efforts on content quality, structure, and the elements that demonstrably affect citations.

    robots.txt: Proven Impact

    Unlike llms.txt, robots.txt configuration has direct, proven impact:

    • Blocking AI crawlers prevents indexing
    • Allowing crawlers enables indexing (though not guaranteed citations)
    • It's respected by all major AI crawlers

    Bottom line: robots.txt configuration is essential; llms.txt is a forward-looking bet.

    Related Articles

    Frequently Asked Questions

    Yes, but for different reasons. robots.txt is essential for controlling crawler access. llms.txt is optional but provides helpful context. They serve complementary purposes.

    Crawlers assume everything is allowed. This is usually fine, but you lose control over blocking sensitive areas.

    Currently, evidence suggests no direct impact. However, the standard is evolving, and early implementation positions you for future benefits.

    robots.txt: When your site structure changes significantly or you want to adjust crawler access. llms.txt: When you add major new content areas, change your site focus, or update key offerings.

    That's a business decision. Blocking prevents training use but also prevents your content from appearing in AI responses. Most businesses benefit more from AI visibility than from blocking.

    For authenticated content, robots.txt matters less since crawlers can't access it anyway. llms.txt could still describe what authorized users can access. Focus efforts on public-facing content.

    No official validator exists. The format is simple Markdown, so any Markdown preview tool can help verify structure. Manual review is recommended.