Most websites have a robots.txt file. It was created in 1994, and webmasters have been updating it ever since to control which crawlers can access which parts of their site. In 2024, a new file emerged that serves a similar purpose for a new generation of visitors: llms.txt.
ChatGPT, Perplexity, Claude, and Google Gemini are now answering hundreds of millions of queries every day. When they answer a question that your content could address, they are making a retrieval decision: which sources should inform this answer, and which should be cited? llms.txt is the file that helps AI systems make that decision in your favor.
As of early 2026, only 5-15% of websites have implemented it. That gap is the opportunity.
What Is llms.txt?
llms.txt is a plain-text Markdown file placed at the root of your website at /llms.txt. It provides a structured, human-readable (and machine-readable) summary of what your site contains and where the most important content lives.
Where robots.txt says "here is what you can and cannot crawl," llms.txt says "here is what we do, here are our most important pages, and here is how to make sense of our content."
The format was proposed by Jeremy Howard (co-founder of fast.ai) in late 2024 and has since been adopted by companies including Anthropic, Cloudflare, Stripe, and Vercel. It is not a Google ranking signal (Google's own crawlers do not use it), but it is increasingly used by the AI retrieval systems that now send high-converting referral traffic: ChatGPT converts at 15.9% and Perplexity at 10.5%, both higher than average organic search.
Why llms.txt Matters for AI Search Visibility
AI language models do not read your website the way a human does. When an AI engine retrieves content to answer a query, it is working from an index built by its crawlers, combined with signals about which sources are authoritative and relevant.
The problem is that most websites were not designed with AI retrieval in mind. A typical company homepage is structured for human readers: hero sections, visual layouts, marketing copy, and CTAs. None of that structure is useful to an AI crawler trying to understand what the company does and what its most citable content is.
llms.txt solves this in three ways:
It provides immediate context. An AI crawler that finds your llms.txt file gets a concise, structured description of your company, products, and content before it reads anything else on your site. This front-loads the signals that determine whether your content is relevant to a given query.
It surfaces your best content. Your most important pages may not be the ones that rank highest in search or attract the most inbound links. llms.txt lets you explicitly point AI systems to the pages you want cited: product pages, in-depth guides, data-backed articles, author profiles.
It signals AI readiness. Sites that have invested in llms.txt signal to AI retrieval systems that their content is maintained, current, and structured for machine consumption. This is a soft trust signal, but at current adoption rates, it differentiates you from the vast majority of your competitors.
Brands that have implemented llms.txt report 2.3x higher AI recall rates (how often their brand appears in AI-generated answers for relevant queries) and a 40-65% increase in AI-referred traffic within six months of implementation.
The llms.txt Format: What Goes in the File
llms.txt uses Markdown formatting. The specification is intentionally minimal: a level-one heading with your site name, a blockquote with a concise site description, and then Markdown sections linking to your most important content.
Here is the complete structure:
# Your Company Name
> One-paragraph description of what your company does, who you serve,
> and what makes your offering distinct. Write this for an AI system
> that has never heard of you. Be specific: name your product category,
> your target customer, and your key differentiators.
## Section Name
- [Page Title](https://yourdomain.com/page/): Optional one-line annotation explaining what this page covers and why it matters.
## Another Section
- [Page Title](https://yourdomain.com/another-page/): Annotation.
The annotations after each URL are optional but valuable. They give AI systems additional context about what a page covers, which improves the likelihood that the page gets retrieved for relevant queries.
Step-by-Step Implementation
Step 1: Write your site description
The blockquote at the top of llms.txt is the most important element. This is the paragraph an AI system reads to decide whether your site is relevant to a query. Write it as a factual, information-dense summary:
- State what your company does in the first sentence
- Name your product category explicitly
- Name your target customer
- List your key differentiators (certifications, pricing model, deployment approach)
- Avoid marketing language; write for comprehension, not persuasion
Example:
> lowtouch.ai is an enterprise AI automation platform that deploys production-ready
> AI agents in 4-6 weeks. Private-by-architecture, no-code configuration, Human-in-the-Loop
> controls on all agentic workflows. SOC 2 Type II and ISO/IEC 27001:2022 certified.
> Outcome-based contracts. Built for CTOs, CIOs, and IT leaders at Enterprises,
> GCCs, and IT Services companies.
Step 2: Choose your sections
Organize your links into logical sections that reflect your site's content architecture. Common sections include:
- Company — about page, team page, contact
- Products / Platform — your main product and feature pages
- Solutions — use-case or vertical-specific pages
- Knowledge Base / Blog — your most important articles and guides
- Key Facts — a bullet list of factual claims AI engines can cite directly
The Key Facts section is particularly useful. It is a direct-citation target: when someone asks an AI "how long does it take to deploy lowtouch.ai?" the answer lives in the Key Facts section of the llms.txt file and can be cited verbatim.
Step 3: Select pages to include
You do not need to list every page on your site. llms.txt is a curated index, not a sitemap. Prioritize:
- Pages that answer questions your target customers ask
- Pages with the most substantive, original content
- Author profile pages (E-E-A-T signal)
- Pages that define your core product or methodology
- Your best blog posts or guides on topics you want to be cited for
Aim for 15-40 URLs across your sections. Too few and you are not giving AI crawlers enough to work with. Too many and you dilute the signal of your most important pages.
Step 4: Write annotations for high-priority pages
Add a colon after each URL and write one sentence explaining what the page covers. Focus annotations on your most important pages; it is fine to leave lower-priority links unannotated.
Without annotation:
- [AMS Payment Issue Monitor](https://www.lowtouch.ai/agents/ams-payment-issue-monitor/)
With annotation:
- [AMS Payment Issue Monitor](https://www.lowtouch.ai/agents/ams-payment-issue-monitor/): AI agent that monitors AMS billing queues, detects payment exceptions, and routes issues to the correct resolution workflow with HITL approval gates.
The annotated version gives AI systems a precise, citable description of the page's content before they even retrieve it.
Step 5: Place the file and set headers
Save your file as llms.txt (not llm.txt, llms.md, or any variation) and place it at the root of your domain: https://yourdomain.com/llms.txt.
Set two HTTP response headers on the file:
Content-Type: text/markdown; charset=utf-8
X-Robots-Tag: noindex
The Content-Type header tells AI agents they are receiving Markdown. The X-Robots-Tag: noindex prevents the file from appearing in Google search results; it is intended for AI systems, not human searchers, so you do not want it indexed.
For most web servers and CDNs, this is a one-line configuration change. For Next.js, add a header rule in next.config.mjs:
async headers() {
return [
{
source: '/llms.txt',
headers: [
{ key: 'Content-Type', value: 'text/markdown; charset=utf-8' },
{ key: 'X-Robots-Tag', value: 'noindex' },
],
},
]
}
Step 6: Keep it current
llms.txt is only as useful as it is accurate. Add new pages when you publish them. Remove pages that no longer exist. Update the site description when your product or positioning changes. Treat it the same way you treat your sitemap: it should always reflect the current state of your site.
Complete Template
Copy this template and fill in your details:
# [Company Name]
> [Company Name] is a [product category] that [core value proposition].
> [Key differentiator 1]. [Key differentiator 2]. [Key differentiator 3].
> Built for [target customer description].
## Company
- [About Us](https://yourdomain.com/about/): [What this page covers]
- [Careers](https://yourdomain.com/careers/): Open roles and company culture
- [Privacy Policy](https://yourdomain.com/privacy-policy/)
- [Terms of Use](https://yourdomain.com/terms-of-use/)
## Platform
- [Homepage](https://yourdomain.com/): [Core product description]
- [Platform Overview](https://yourdomain.com/platform/): [Feature summary]
## Products / Agents / Solutions
- [Product 1](https://yourdomain.com/products/product-1/): [What it does]
- [Product 2](https://yourdomain.com/products/product-2/): [What it does]
## Knowledge Base
- [Article Title](https://yourdomain.com/article-slug/): [What the article covers]
- [Guide Title](https://yourdomain.com/guide-slug/): [What the guide covers]
## Key Facts
- Pricing: [pricing model description]
- [Certification or credential]: [details]
- [Key metric or claim]: [details]
- Contact: [contact email or URL]
Common Mistakes
Listing every page instead of curating. llms.txt is a signal of what matters, not a complete inventory. Including 200 URLs dilutes the importance of your critical pages. Keep it focused.
Writing the description as marketing copy. "We are the world's leading AI platform" tells an AI system nothing useful. "We deploy private, no-code AI agents for enterprise IT operations teams in 4-6 weeks" is specific, factual, and citable. Write for comprehension.
Forgetting to update it. A llms.txt file that references pages you have deleted or products you no longer offer signals to AI systems that your content may be stale. Set a calendar reminder to review it quarterly.
Using the wrong filename. The specification requires exactly llms.txt. Variations like llm.txt or llms.md are not recognized by AI retrieval systems that follow the standard.
Skipping annotations on key pages. The annotation is where you control how AI systems describe your content when they cite it. A bare URL link is better than nothing; an annotated URL is significantly more effective.
What llms.txt Does Not Do
It is worth being clear about the limits of llms.txt to avoid over-indexing on it at the expense of more foundational work.
It is not a Google ranking signal. Google's own crawlers do not use llms.txt. It will not improve your position in traditional search results.
It does not override robots.txt. If your robots.txt blocks a crawler from accessing your content, llms.txt cannot override that block. AI crawlers that follow robots.txt (most do) will still be unable to retrieve blocked pages even if they appear in llms.txt. Fix your robots.txt first.
It is not a substitute for quality content. llms.txt helps AI systems find and contextualize your content. It does not make weak content worth citing. The underlying content still needs to be accurate, structured, and authoritative.
It works alongside, not instead of, technical SEO. Fast page load, proper schema markup, named authors, and clean canonical tags all contribute to AI citation rates independently of llms.txt. The file accelerates discovery and context; the rest of your technical SEO determines whether the content is worth citing once found.
Verifying Your Implementation
After publishing your llms.txt file, verify it with two checks:
Accessibility check: Visit https://yourdomain.com/llms.txt in a browser. Confirm the file loads as plain text and includes your full content. Check the response headers using browser developer tools or curl -I https://yourdomain.com/llms.txt to confirm Content-Type and X-Robots-Tag are set correctly.
AI readiness audit: Run your domain through seo.yatna.ai's AI readiness audit. The AI Crawler Access category (20% of the total score) checks for llms.txt presence and validates its format alongside your robots.txt configuration. It takes under five minutes and identifies any gaps in your broader AI search readiness setup.
The Adoption Curve: Why Acting Now Matters
Of the 844,000 websites that have already implemented llms.txt, the majority are in the technology, SaaS, and developer tools categories. Outside those niches, adoption is still well under 5% across most industries.
This means that in most verticals, you can establish topical authority with AI retrieval systems before your competitors do. The sites that implement llms.txt today are building an association between their brand and their core topics that will persist as AI search adoption accelerates.
The file takes under an hour to write. The competitive advantage it creates compounds over time as AI systems become more reliant on structured signals to navigate the growing volume of web content.
Start with the template above. Add your 20 most important pages. Publish it today.
Dr. Anil Kumar is VP of Engineering at lowtouch.ai, with over 25 years of experience in solution architecture and enterprise IT. He leads engineering initiatives across lowtouch.ai's AI agent platform and AI search tooling. Audit your site's AI search readiness at seo.yatna.ai.
