Why Do We Now Add llms.txt to Every Website We Build for Clients?
Last month, we received surprising feedback from an operations manager at one of our B2B foreign trade company clients:
"Besides traditional Google and backlink sources, we're now getting traffic from GPT."
Beyond SEO: Enter AI LLM EO
In this increasingly competitive era, both we and our clients are constantly thinking about how to increase visibility to potential customers.
AI large language models have become a new way for people to find information, get answers, and interact with the world:
- Many people now turn first to Deepseek, Claude, or GPT when faced with decisions.
- People aren't just typing keywords into search boxes anymore; they're directly asking AI to summarize, analyze, and provide answers or solutions.
We realized that having a feature-complete website with good SEO (including algorithm optimization, keyword research, site speed improvements, readability, robots.txt, sitemap.xml, etc.) is no longer enough.
We need to ensure AI models can see, index, and optimize our content in a way that encourages them to recommend it.
Many brilliant minds have been exploring this challenge.
From the perspective of providing more complete, accurate, and effective information to large models, Jeremy Howard's llms.txt proposal represents a highly viable solution from the supply side.
Most importantly, large language models and applications with web search capabilities like Anthropic (Claude) and Perplexity are gradually beginning to support llms.txt.
After internal discussions, we made a decisive move: from now on, our team will add llms.txt to all new websites we build, and we'll also incorporate it into existing site maintenance (at no additional cost)!
From "Passive Defense" to "Active Engagement"
Our websites (whether they're standalone foreign trade sites, corporate websites, personal blogs, or SaaS platforms) need to appeal not just to search engines, but also to AI (or rather, receive better AI evaluations).
Traditional websites contain what AI considers "noise" and "junk information" - flashy animations, various format tags, technical information from development frameworks, and potentially misleading low-relevance advertisements.
AI has to work hard to analyze and filter this content, and might even "hallucinate" unreliable information, leading to inaccurate answers. llms.txt is an optional solution that helps AI understand our clients' websites more efficiently and accurately.
It's more than just a technical file - it's key to establishing an "official hotline" between websites and AI models! It transforms our approach from "passively waiting for AI to read/analyze/understand" to "actively telling AI what we have," guiding it to precisely understand and present our clients' brand information.
How Does It Differ from robots.txt and sitemap.xml?
| Filename | Core Purpose | Primary Audience |
|---|---|---|
robots.txt | Manage Crawler Behavior: Tells search engine crawlers (like Googlebot, Baiduspider) which pages to crawl and which to avoid, preventing server overload and protecting private content. | Traditional Search Engine Crawlers |
sitemap.xml | Provide Site Map: Lists all indexable page URLs, helping search engines discover and crawl website content more comprehensively, especially beneficial for new sites or frequently updated content. | Traditional Search Engine Crawlers |
llms.txt | Optimize AI Understanding: Provides core, high-quality, clean content guidance for Large Language Models, helping AI understand website information more efficiently and accurately while reducing hallucinations and false reports. | Large Language Models (LLMs) and their applications (like AI chatbots, AI search engines) |
robots.txt, sitemap.xml, and llms.txt are all crucial bridges between websites and the external "robot" world. robots.txt sets "boundaries," sitemap.xml provides a "map," and llms.txt offers an "essential guide."
Configuring these "three musketeers" properly will be our default approach when developing public-facing websites for clients.
The goal is to ensure website content can be efficiently and accurately discovered, indexed, and understood by both search engines and AI models, maximizing value across different information consumption scenarios.
How Do We Integrate llms.txt into Our Development Process?
Implementing llms.txt isn't as simple as generating a random file. It requires ongoing collaboration between both parties, deep understanding of client businesses, and rethinking content presentation. Here's our typical approach:
-
Content "Spring Cleaning" and "Strategic Deployment": At project start, we sit down with clients to review core content, key product/service information, brand stories, and technical documentation. We guide clients to identify which information they most want AI to accurately convey to users. This isn't just a technical exercise - it's an opportunity to refine the brand's core values.
-
High-Quality Markdown Content "Production Line": The essence of
llms.txtis providing high-quality Markdown content. We help clients transform their core information into concise, clear, no-fluff Markdown format. This includes product introductions, API documentation, FAQs, company profiles, etc. We ensure the Markdown content is both AI-efficient and human-readable. -
llms.txt File "Fine-Tuning": Once content is ready, we structure the file according to
llms.txtconventions. We skillfully use H1, H2 headings, blockquotes, and link lists to ensure AI can instantly locate the most important information. Less critical but still useful content goes into the "Optional" section, allowing AI to skip it during quick reads. -
"After-Sales Service" and "Continuous Updates":
llms.txtisn't a one-and-done solution. As client businesses evolve and website content updates,llms.txtmust evolve too. We provide detailed maintenance guidelines and tool support to ensurellms.txtstays current and optimal.
For example, a SaaS company's core product might include both a user-facing web interface and complex API services for technical users. A well-configured llms.txt provides detailed API documentation in Markdown format. Ideally, developers can quickly find solutions through AI programming assistants using accurate information from llms.txt. This not only enhances user experience but also significantly reduces customer support workload - a win-win situation.
In our next article, we'll share our technical notes about llms.txt, including how to implement static/dynamic llms.txt in various technical frameworks like WordPress/React/Vue, continue sharing our technical insights about llms.txt, and explore interesting llms.txt implementations from other websites!
About Us: We are a professional technology services company. We provide small but beautiful tools/systems focused on quickly solving specific problems, covering solution consulting, software development, system integration, and AI solutions. Building official websites, landing pages, and standalone sites is an important part of our extended services. Specifically including:
About Us:
- IT Services:
- Custom Development (customizing highly available tools or systems for business pain points)
- System Integration (seamlessly connecting new and old systems with internal and external SaaS tools to achieve cross-platform business flow automation)
- Information Digitalization Consulting (providing targeted consulting services based on digital efficiency industry practices and technical solution implementation experience)
- AI Services:
- AI Workflow Automation (embedding AI decisions in business flows, such as inquiry→CRM automation, reducing end-to-end processing time and labor costs by over 60%)
- Multimodal AI Insights (AI + text, image, audio, video, from files to data to information, discovering hidden business value)
- AI Knowledge Base Driven Content Creation (generating social media posts/marketing copy/product documentation based on industry knowledge base and business insights) Feel free to contact us to discuss your customization needs and receive your personalized solution.
Cover Image by wordgardener from Pixabay.