llms.txt: What It Is and How to Implement It
llms.txt is a plain-text file placed in the root directory of your website that gives large language models (LLMs) a structured, curated summary of your most important content. It was proposed by fast.ai as a lightweight convention to help AI systems understand what your site is about without having to crawl and interpret every page independently.
If you want AI-powered search engines and assistants to represent your business accurately, understanding and implementing llms.txt is one of the most direct technical steps you can take.
Why llms.txt Exists
The Problem With How AI Systems Currently Read Websites
Large language models do not read websites the way a human does. When an AI crawler visits your site, it processes raw HTML, strips out navigation, scripts, and boilerplate, and tries to extract meaning from what remains. That process is imperfect. Important context gets lost, key claims get misattributed, and your brand narrative can end up distorted by the time it reaches a user through an AI-generated answer.
Traditional web infrastructure was built for human readers and, later, for Google-style keyword-matching crawlers. Neither was designed with conversational AI in mind.
The Solution: A Machine-Readable Summary Built for LLMs
llms.txt addresses this by giving you direct control over the summary an AI model receives. Instead of reverse-engineering your site structure, a language model can retrieve a single, well-formatted file that tells it:
- What your organisation does
- What your most important pages are
- How those pages relate to each other
- Any specific guidance for how that content should be interpreted
Think of it as a brief for the AI, written by you, in plain language, structured for optimal consumption.
What Is llms.txt? A Technical Definition
llms.txt is a Markdown-formatted plain-text file served from the root of your domain at https://yourdomain.com/llms.txt. It follows a loosely defined structure that includes:
- A top-level heading containing your site or organisation name
- A short blockquote summarising what the site does (this acts as the primary description)
- Optional detail sections providing additional context about the organisation, product, or service
- A series of Markdown lists linking to the most important pages on your site, grouped by topic or content type
- Your homepage
- Your core product or service pages
- Your most authoritative blog posts or guides
- Your about page
- Any high-value case studies or evidence pages
- Service Page One: Brief description of what this page covers.
- Service Page Two: Brief description of what this page covers.
- Guide Title: What this guide explains and who it is for.
- Glossary Term: Definition and context.
- Terms of Service: Legal terms for use of this site.
- Privacy Policy: How user data is handled.
- WordPress: Upload via FTP/SFTP to the
public_htmldirectory, or use a plugin that supports custom root-level files. - Webflow: Use the
publicfolder in project settings to add custom files. - Shopify: Use the
config/directory or a custom app to serve files from root. - Static site generators (Next.js, Gatsby, Astro): Place the file in the
publicfolder of your project. - Launch new core service pages
- Publish significant guides or reports
- Restructure your site navigation
- Update your positioning or description
The file is intentionally minimal. It is not a sitemap. It is not a replacement for structured data. It is a concise, human-readable brief that an LLM can process in a single context window.
What llms-full.txt Is
An optional companion file, llms-full.txt, concatenates the full text of all pages listed in llms.txt into one document. This is intended for AI systems that prefer to ingest bulk content rather than follow links. Not all implementations include this file, and it is only worth producing if your content is genuinely valuable in full rather than in summary form.
How llms.txt Differs From Other Technical Files
llms.txt vs robots.txt
robots.txt is a permissions file. It tells crawlers which parts of your site they are allowed to access. llms.txt carries no permissions or restrictions at all. It is purely informational, a curated guide rather than a gatekeeper.
You can, and should, have both. They serve entirely different purposes.
llms.txt vs sitemap.xml
A sitemap lists every URL on your site so that search engine crawlers can discover and index pages efficiently. llms.txt is selective. You include only the pages that best represent your content and authority. The aim is quality of signal, not completeness of coverage.
llms.txt vs Structured Data (Schema Markup)
Schema markup is embedded within individual page HTML and helps search engines understand the type and properties of content on that specific page. llms.txt operates at the site level, providing a holistic summary rather than page-level metadata. The two are complementary and both matter for AI search optimisation.
How to Implement llms.txt
Step 1: Plan Your Content Hierarchy
Before writing the file, identify the pages that best represent your site. For most businesses, this will include:
Aim for quality over quantity. A well-chosen list of ten pages is more useful to an LLM than an exhaustive catalogue of fifty.
Step 2: Write the File in Markdown
The structure of a valid llms.txt file looks like this:
# Your Organisation Name
> A one or two sentence description of what your organisation does and who it serves.
About
Additional context about your organisation, mission, or unique positioning. Keep this concise.
Core Services
Key Resources
Optional
Every line item in the lists should include a link and a brief description. The description is important: it tells the LLM what to expect from the linked page before it decides whether to retrieve it.
Step 3: Save and Deploy the File
Save the file as llms.txt (no subdirectory, no variation in filename) and deploy it to the root of your domain. The file must be publicly accessible at https://yourdomain.com/llms.txt.
For most CMS platforms:
Step 4: Verify Accessibility
Once deployed, visit https://yourdomain.com/llms.txt in a browser. You should see the raw Markdown text. If you see a 404 error, the file is not in the correct location or your server is not configured to serve plain-text files from root.
You can also use a tool such as curl from the command line:
curl https://yourdomain.com/llms.txt
A successful response will return the file contents with a 200 OK status code.
Step 5: Keep It Updated
llms.txt is only useful if it reflects your current content. Set a reminder to review and update the file whenever you:
Stale content in llms.txt can mislead AI systems just as readily as stale content anywhere else on your site.
Common Mistakes to Avoid
Using It as a Promotional Document
llms.txt should be descriptive, not persuasive. Language models are not persuaded by marketing copy. They are looking for accurate, factual descriptions of what pages contain. Treating the file as advertising copy reduces its utility.
Listing Too Many Pages
Including every page on your site defeats the purpose. The value of llms.txt comes from curation. If you list 200 pages, an LLM has to do nearly as much work as if it crawled the site itself.
Ignoring the Descriptions
Links without descriptions are significantly less useful. A URL alone gives an LLM no signal about whether that page is worth retrieving. Always include a clear, accurate one-sentence description for each item.
Forgetting HTTPS and Canonical Consistency
Ensure all URLs in your llms.txt file use HTTPS and match the canonical versions of your pages. Inconsistencies between llms.txt links and your canonical tags can create conflicting signals for AI systems.
llms.txt and Your Broader AI Search Strategy
Implementing llms.txt is one component of a complete AI search optimisation strategy. On its own, it helps AI systems understand your site structure. Combined with well-structured page content, appropriate schema markup, and a strong E-E-A-T profile, it significantly improves the likelihood that AI-powered tools accurately cite and represent your business.
It is worth noting that major AI platforms such as Perplexity, ChatGPT with browsing, and Google's AI Overviews do not officially publish documentation stating they use llms.txt. However, the file costs almost nothing to implement, carries no downside risk, and provides a clear signal of intent to any AI system that does retrieve it.
For businesses investing in being found through AI search, the question is not whether llms.txt alone will transform your visibility. The question is whether you have done everything reasonable to make your content as accessible and interpretable as possible. llms.txt is one of those reasonable steps.
Get Your AI Search Presence Audited
Not sure how well AI systems currently understand your website? Our free audit reviews your site's AI search readiness, including your llms.txt implementation, structured data, and content signals.
Request your free AI search audit at beknown.world/audit
Frequently Asked Questions
<FAQPage items={[
{
question: "What is llms.txt?",
answer: "llms.txt is a plain-text file placed in the root directory of a website that provides structured, concise information about a site's content specifically for large language models and AI-powered search engines to consume."
},
{
question: "Is llms.txt an official standard?",
answer: "No. As of early 2025, llms.txt is a proposed convention rather than a ratified web standard. It was introduced by fast.ai and has gained traction in the AI search community, but it has not been formally adopted by a standards body."
},
{
question: "What is the difference between llms.txt and robots.txt?",
answer: "robots.txt tells crawlers which pages they are or are not permitted to access. llms.txt does not restrict access; instead, it provides a curated, human-readable summary of your site's most important content specifically formatted for language models."
},
{
question: "Does llms.txt improve my ranking in AI search engines?",
answer: "Implementing llms.txt will not guarantee higher rankings, but it increases the likelihood that AI systems accurately understand and represent your content. Accurate representation often correlates with being cited as a source in AI-generated answers."
},
{
question: "How do I verify that AI crawlers are reading my llms.txt file?",
answer: "Check your server access logs for requests from known AI crawler user agents such as GPTBot, ClaudeBot, and PerplexityBot. If they are crawling your root directory, they will typically retrieve llms.txt as part of that process."
},
{
question: "Can I have both llms.txt and llms-full.txt?",
answer: "Yes. The convention supports two variants. llms.txt is a short summary file with links to key pages. llms-full.txt is an optional expanded version that concatenates the full content of those linked pages into a single document for models that prefer bulk ingestion."
},
{
question: "What happens if I do not implement llms.txt?",
answer: "AI systems will still crawl and index your site using other signals, but they will piece together an understanding of your content without the structured guidance you could provide. This increases the chance of misrepresentation or omission in AI-generated answers."
}
]} />