Google Says Ignore llms.txt for SEO — But Lighthouse Disagrees

Google’s own products are now contradicting each other on llms.txt, and the split has direct operational consequences for any marketing team managing AI-era web presence. As reported by Search Engine Journal on May 20, 2026, Google Search explicitly tells webmasters the file is unnecessary for AI Overviews and AI Mode, while Lighthouse 13.3’s new Agentic Browsing audit flags the file’s absence as a signal of poor agentic readiness. If you’re responsible for organic search, AI agent visibility, or technical SEO strategy, you’re now dealing with two different answers from the same company — and you have to pick which one actually governs your build priorities.

What Happened

The conflict surfaced publicly when Search Engine Journal reported that Google’s guidance on llms.txt varies depending on which product team you consult. The two positions could not be more clearly opposed: the Search team says the file is irrelevant, while the Chrome team has baked a llms.txt presence check into an official developer tool audit.

Google’s Search Relations lead John Mueller previously compared llms.txt to the keywords meta tag — a file that sounds useful in theory but that no major AI service actively uses for ranking or content discovery. That framing was precise and deliberate. The keywords meta tag is the canonical SEO cautionary tale about a well-intentioned signal that got deprecated into irrelevance; invoking it was Mueller’s way of warning practitioners off the file before adoption scaled. The comparison has stuck, and Google Search’s official documentation reinforces it consistently.

The Google AI Overviews and AI Mode guidance states directly: “There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.” It continues: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features. There’s also no special schema.org structured data that you need to add.” At Search Central Live Deep Dive Asia Pacific, Google’s Gary Illyes and Amir Taboul confirmed in person: Google was not pursuing llms.txt. Three separate Google Search signals — Mueller’s public comments, the official documentation, and a live conference confirmation from two Google engineers — all land on the same conclusion.

Then there’s Lighthouse. With version 13.3, Google’s own open-source performance and quality auditing tool shipped a new category: Agentic Browsing audits. Unlike the traditional 0–100 performance score, the Agentic Browsing category uses a fractional pass/fail model — it tallies which readiness checks your site passes out of the total. One of those checks is llms.txt presence, verified at the domain root. The Chrome Developer documentation describes llms.txt as providing “a machine-readable summary” that helps LLMs and agents understand site structure. It notes that “without the file, agents may spend more time crawling” — which is not nothing when agent time and token consumption translate directly to user experience latency and infrastructure cost.

The contradiction got stranger in December 2025, when llms.txt files quietly appeared on several Google developer properties: Search Central, developer.chrome.com, and web.dev. The Search Central file was pulled within hours of its appearance. That rapid removal, combined with the fact that Chrome and web.dev kept their files live, suggested the deployment was an automated CMS rollout — probably a template change that applied across Google’s developer property network — rather than a deliberate Search team endorsement. But the fact that the Chrome team left their file up while Search deleted theirs is a concrete expression of the internal disagreement in real-world infrastructure decisions.

The core issue is that these are genuinely separate use cases that happen to share a file format. Google Search guidance addresses what affects search ranking and AI Overviews inclusion — a crawler-driven, index-based system where Googlebot processes content asynchronously. Lighthouse’s Agentic Browsing audits address something different: whether a browser-based AI agent can efficiently navigate and understand your site during an active, live session. One is about being found by an index; the other is about being efficiently usable by an agent in real time. The confusion for practitioners arises precisely because the distinction isn’t prominent in either team’s communications, leaving marketing teams to navigate contradictory signals from what appears to be a single authoritative source.

Why This Matters

The practical impact of this divergence depends on what kind of AI visibility you’re optimizing for — and most marketing teams are not yet clearly distinguishing between the two underlying channels that produce these conflicting answers.

Google Search AI Overviews operates through Google’s existing crawl-and-index infrastructure. Googlebot processes your content, it enters the index, and the AI summarization layer draws from that indexed corpus when generating Overviews. For this channel, the Search team’s guidance holds without qualification: llms.txt does not influence whether or how your content surfaces in AI Overviews. Your existing SEO fundamentals — crawlability, indexability, E-E-A-T signals, structured and well-organized content — are what drive performance here. Spending engineering cycles on llms.txt in pursuit of AI Overviews visibility is misdirected effort, full stop.

Agentic browsing is a different channel entirely. When a browser-based AI agent — whether that’s Google’s own emerging agentic products, OpenAI’s Operator, Anthropic’s Claude with computer use capabilities, or any number of third-party automation agents — navigates to your site to complete a task, it doesn’t work through an index. It operates in real time, reading your actual rendered page content, your accessibility tree, and now potentially your llms.txt to orient itself quickly before engaging with your site structure. Lighthouse’s agentic audits evaluate your readiness for this live-session interaction model, which is mechanically different from search at every layer of the stack.

For marketing teams, this creates a workflow split that currently has no clean resolution from Google:

Agencies running technical SEO audits now face a checklist conflict. Standard practice is to align with Google Search guidance, which is well-documented and consistently communicated. But if you’re delivering AI-ready site audits and you ignore Lighthouse’s agentic readiness criteria, you’re producing incomplete deliverables for clients who are paying for a full view of AI optimization. The billable question becomes: which Google are you optimizing for, and are you telling the client that these are different answers?

In-house SEO teams at enterprise brands face the same conflict with higher internal stakes. Enterprise sites have layers of stakeholders with opinions on technical changes. Introducing llms.txt now — when the Search team explicitly says it’s unnecessary — requires a clear rationale that most internal teams can’t easily communicate upward without generating confusion. But failing to implement it while agentic browsing adoption accelerates means a retroactive remediation project at exactly the wrong time — when the channel is material and remediation has business urgency.

Solopreneurs and small teams running lean stacks may find the easiest path is simply to implement the file once at low cost and remove it from the decision queue. The implementation cost is modest — a few hours to draft, format, and deploy a static markdown file. The risk of not having it is speculative today but increasingly concrete as browser-agent traffic scales.

Content publishers and media companies face a version of this problem with an added dimension: llms.txt exposes content structure to AI agents in ways that raise legitimate questions about content licensing and usage permissions. Some publishers are actively using the file to communicate what agents are and aren’t permitted to consume — a use case that Google Search’s dismissal of the file entirely sidesteps.

What the divergence is really signaling is that “Google” is not a monolith when it comes to AI optimization guidance. The Search team, the Chrome team, and the Lighthouse team are each building toward different futures of AI-integrated web interaction, and they are not yet speaking with one coordinated voice. Marketers who treat Google as a single unified source of truth on AI optimization strategy are going to get burned by whichever team’s vision turns out to govern the channel that matters most to their business.

The Data

The following table maps each Google product’s current documented position on llms.txt and what that means for marketing practitioners:

The following table maps the llmstxt.org specification — proposed by Jeremy Howard in September 2024 — against Lighthouse Agentic Browsing audit requirements:

The llmstxt.org specification has already spawned CMS plugins for VitePress, Docusaurus, and Drupal, as well as CLI tools, JavaScript and PHP libraries, and community directories cataloging live implementations — indicating the format has real momentum in the developer and technical marketing community regardless of Google Search’s position on it.

Real-World Use Cases

Use Case 1: E-Commerce Brand Preparing for AI Shopping Agents

Scenario: A mid-market e-commerce brand selling outdoor gear begins seeing early traffic signals from browser-based shopping agents — users are delegating product research and checkout initiation to AI agents that browse on their behalf. Their product pages are well-optimized for Google, but agent sessions show high task abandonment and poor completion signals. The team knows this traffic is small today but wants to address it before it becomes material.

Implementation: The brand drafts an /llms.txt file structured per the llmstxt.org spec: an H1 with their brand name and product focus, a blockquote briefly describing the catalog structure, and H2-delimited sections linking to their category index pages, the brand’s structured product data endpoints, shipping and returns policy in plain text, and a size guide formatted in markdown. They run Lighthouse Agentic Browsing audits against their homepage and top category pages and find two additional failures: CLS issues from an image carousel component and missing ARIA labels on filter controls. Both get added to the next sprint. The llms.txt file is deployed as a static file served from the CDN root.

Expected Outcome: AI agents navigating the site orient more quickly and reach relevant product pages with fewer crawl steps. Task completion rates for agent-driven sessions improve measurably. The brand also establishes an agentic readiness baseline score in Lighthouse they can track as a KPI alongside Core Web Vitals.

Use Case 2: SaaS Company Controlling AI Agent Context for Sales Research

Scenario: A B2B SaaS company selling marketing analytics software knows that sales prospects increasingly use AI assistants to research vendors before requesting demos. They want to ensure that when agents research their product, those agents surface accurate current pricing and positioning — not outdated cached content, competitor-framed comparisons, or blog posts from two product generations ago.

Implementation: The team builds an /llms.txt file that explicitly links to their current pricing page, their case study library, their technical documentation index, and a clean-markdown product overview. They deliberately exclude the blog and news section from the curated links — too much historical noise. They include their comparison landing page where they address competitor differentiators on their own terms. Using the llmstxt.org spec‘s Optional section, they flag their release notes and changelog as secondary content agents can skip in constrained contexts. They add a last-updated timestamp to the blockquote so agents can assess content freshness before relying on it.

Expected Outcome: AI assistants used by prospects during vendor research are more likely to surface accurate current pricing, relevant case studies, and the team’s preferred positioning framing. Sales teams report fewer discovery calls where prospects bring outdated information into the conversation. Implementation cost is approximately four hours upfront and quarterly maintenance tied to the company’s existing content refresh cycle.

Use Case 3: Digital Agency Updating AI Readiness Audit Deliverables

Scenario: A digital agency with a strong SEO practice has been delivering AI readiness audits that follow Google Search guidance — which means not recommending llms.txt. A sharp client reads the Lighthouse 13.3 release notes and asks why their recent audit didn’t address the new Agentic Browsing category. The agency needs to update its audit framework and client-facing documentation without appearing to contradict the Google Search guidance they’ve been accurately citing for months.

Implementation: The agency updates its audit template to explicitly distinguish two optimization layers with clear language for client presentations: (1) Search AI Layer — AI Overviews and AI Mode visibility, governed by Google Search guidelines where llms.txt is a confirmed non-factor; and (2) Agentic Browsing Layer — browser-based agent session readiness, governed by Lighthouse Agentic Browsing audits where llms.txt presence is a measured signal alongside ARIA labeling, CLS, and WebMCP integration. They build a standard llms.txt template into their onboarding deliverable package and add Lighthouse Agentic Browsing fractional scores to their monthly performance dashboard as a tracked metric separate from Core Web Vitals.

Expected Outcome: Clients receive a complete, accurate picture of AI optimization across both channels. The agency differentiates its practice by demonstrating fluency with emerging agentic standards while maintaining precise Google Search guidance — and the client who asked the pointed question becomes a reference account for the updated framework.

Use Case 4: Content Publisher Communicating AI Access Permissions

Scenario: An independent media publisher with a high-authority editorial site is actively managing AI content consumption. They already block specific AI crawlers via robots.txt for training data purposes, but they want a more nuanced layer that communicates usage permissions to AI agents that do have access — specifically, agents performing research and summarization tasks on behalf of readers.

Implementation: The publisher deploys an /llms.txt file that structures permitted content categories explicitly: public news summaries, evergreen explainer articles, and product reviews are listed with direct links. Paywalled investigative reports and proprietary research are excluded entirely from the curated file. The blockquote includes a plain-language usage note indicating that content may be referenced with attribution but not reproduced in full. Using the llmstxt.org spec‘s Optional section, they flag lower-priority archival content, steering agents toward their most current and authoritative material rather than outdated archives.

Expected Outcome: AI agents that parse the file structure prioritize the publisher’s editorial summaries and prefer attribution-compatible usage patterns. The publisher has a documented, machine-readable record of their stated AI access policy — which carries increasing weight as content licensing for AI systems becomes a more active regulatory and legal concern across jurisdictions.

Use Case 5: Enterprise Brand Tracking Agentic Readiness Across a Large Site

Scenario: An enterprise consumer brand with a large, complex site — thousands of product pages, support documentation, regional microsites, and a retail locator tool — wants to assess agentic browsing readiness systematically before browser-agent traffic becomes a measurable channel in their analytics. They want a framework and a baseline, not a scramble.

Implementation: The team runs Lighthouse Agentic Browsing audits across a representative set of their highest-traffic page templates: homepage, product detail page, support article, and store locator. They document current fractional pass scores across all four Agentic Browsing audit areas: WebMCP integration, agent accessibility (ARIA labeling and tree integrity), layout stability (CLS), and discoverability (llms.txt). Results show that support content performs well on accessibility but the retail locator tool fails on both CLS and llms.txt. They deploy a root-level /llms.txt covering their main site — product catalog, support index, brand overview — and schedule a separate remediation sprint for the locator tool’s layout stability issues. They establish a quarterly Lighthouse Agentic Browsing audit cadence against the same template set and add the fractional score to their quarterly technical SEO review.

Expected Outcome: The enterprise has a documented agentic readiness baseline, a prioritized remediation roadmap tied to real audit scores, and a governance model for tracking improvement over time. When browser-agent-driven sessions become measurable in their analytics, remediation is already underway rather than just beginning.

The Bigger Picture

The llms.txt disagreement between Google’s Search and Chrome teams isn’t a communications failure — it’s a structural symptom of how quickly the web’s interaction model is fragmenting, and how early we still are in understanding what “AI optimization” actually means across different channels.

For most of the web’s history, Google was the dominant consumer of web content at scale, and SEO was the discipline of optimizing for one large crawler with a known and documented signal set. That model is over. Today’s web is consumed by at minimum three distinct AI interaction models operating simultaneously: asynchronous index-based AI search (Google AI Overviews, Bing Copilot), real-time browser-based AI agents that navigate live sites to complete tasks on behalf of users, and direct LLM inference where models are queried about brands and products based on training data. Each of these channels has different optimization requirements, different mechanics, and — as Google is demonstrating internally — different teams with different priorities and technical approaches.

The llmstxt.org specification, proposed by Jeremy Howard in September 2024, was designed for the second and third models specifically. Howard’s stated rationale was precise: “context windows are too small to handle most websites in their entirety,” so a curated, structured markdown file gives LLMs and agents a navigable entry point into a site’s content architecture. That rationale is entirely correct for agent sessions and direct LLM queries, and entirely irrelevant for Google’s asynchronous index-based search pipeline — which explains why the Search team is right for their channel and the Chrome team is right for theirs.

What we’re watching is the early phase of a larger standards competition for the agentic web. The llmstxt.org specification has already generated a meaningful ecosystem: CMS plugins for VitePress, Docusaurus, and Drupal; CLI tools; JavaScript and PHP libraries; community directories cataloging live implementations. Anthropic, Perplexity, and a growing number of AI agent frameworks have expressed interest in or varying degrees of support for the format. Google Chrome’s inclusion of llms.txt in Lighthouse — even in an experimental label — is a significant signal that the format is progressing from community proposal toward a de facto standard for the agentic channel, regardless of what the Search team thinks.

The implication for the AI marketing landscape is that “Google SEO” is no longer a unified discipline with a single governing set of guidelines. Marketers will increasingly need to maintain parallel optimization tracks for distinct channels: search AI (governed by Google Search documentation), agentic browsing (governed by Lighthouse and emerging browser-agent specs), and direct LLM inference (currently ungoverned by any major platform). The practitioners and agencies who recognize this fragmentation early and build structured, channel-specific workflows will have a meaningful operational advantage over those treating “AI SEO” as a single, undifferentiated practice to be governed by whichever guidance article came up first in a Google search.

What Smart Marketers Should Do Now

1. Implement llms.txt for agentic readiness — and explicitly frame it that way internally.
The correct framing is not “we’re adding llms.txt to rank better in Google.” The correct framing is “we’re adding llms.txt to perform better in agentic browsing sessions, which is a distinct channel from search.” The answer to the first question is no, per Google Search’s own documentation. The answer to the second is yes, and the implementation is low-cost enough to justify proactive action. Follow the llmstxt.org spec: H1 with your site or brand name, a brief blockquote orientation summary, and H2-delimited curated sections linking to your most important content areas. Keep it focused and current — this is not a sitemap, it’s a structured orientation document for agents navigating your site in real time.

2. Run Lighthouse Agentic Browsing audits on your top page templates now to establish a baseline.
The Agentic Browsing audit is available in Lighthouse now and costs nothing to run. Audit your homepage, your highest-traffic landing pages, and your primary conversion pages. The audit evaluates four areas: WebMCP integration, agent accessibility (proper ARIA labeling and accessibility tree integrity), layout stability (CLS), and discoverability (llms.txt). You don’t need to act on every result immediately, but getting a baseline fractional score today gives you a meaningful benchmark for improvement. Treat the Agentic Browsing score the way early practitioners treated PageSpeed scores when performance metrics first became auditable: establish the baseline, document the failures, and build toward improvement systematically.

3. Update your AI optimization framework to separate search AI from agentic browsing.
This is especially critical for agencies and in-house teams who brief clients or senior stakeholders. Google’s official Search documentation is accurate and should be followed for AI Overviews and AI Mode optimization. But it explicitly and exclusively addresses the search channel. The moment a conversation shifts to agentic browsing, browser-integrated AI, or direct LLM context retrieval, that documentation no longer applies and citing it will generate incorrect recommendations. Build and document a two-track framework — Search AI optimization and Agentic Web optimization — and make the channel distinction explicit in every client conversation, audit deliverable, and internal SOW before the conflation creates a problem.

4. Prioritize your site’s accessibility tree as a structural agentic investment.
The Lighthouse Agentic Browsing audit’s emphasis on ARIA labeling, element naming conventions, and accessibility tree integrity isn’t primarily an ADA compliance concern in this context — it’s a machine readability investment. AI agents use the accessibility tree as their primary structural model of your page. Missing labels, improperly hidden elements, and unlabeled interactive controls are not just accessibility failures for human users; they are agent navigation failures that force agents to spend more time and more tokens developing a working model of your page before they can complete any task. An accessibility audit conducted through the lens of agentic readiness delivers simultaneous returns across human accessibility compliance, agent usability, and Lighthouse Agentic Browsing scoring — three business goals addressed with a single remediation sprint.

5. Assign ownership of llms.txt maintenance and track the spec’s evolution.
The llmstxt.org spec is community-governed and actively developing. The current version supports a primary /llms.txt and an extended /llms-full.txt variant. As more AI agent frameworks formally adopt the standard, additional conventions around usage permissions, content freshness signals, and structured metadata are likely to emerge as community conventions or formal spec extensions. Assign clear ownership of your llms.txt to someone on your technical SEO or web operations team, set a quarterly review cadence tied to your content architecture changes, and monitor the llmstxt.org GitHub and community channels for spec updates. Treat this file the way you treat robots.txt and your XML sitemap — not as a one-time deployment but as a living document that needs to reflect your current site structure and content strategy to remain useful for the agents reading it.

What to Watch Next

Lighthouse Agentic Browsing graduating from experimental status. The current audit is labeled experimental and uses a fractional pass/fail scoring model rather than the traditional 0–100 Lighthouse score. The key milestone to watch is when Google removes the experimental designation and potentially integrates agentic readiness scores into mainstream performance reporting products — PageSpeed Insights, Search Console, or CrUX. When Core Web Vitals moved from a Lighthouse flag to a confirmed ranking signal, it catalyzed an entire industry sprint on performance optimization. A similar transition for Agentic Browsing scoring would have an equivalent forcing function for agentic readiness investment. Watch Chrome DevRel blog announcements and PageSpeed Insights release notes for this transition, most likely in Q3 or Q4 2026.

Third-party AI agent platforms formalizing llms.txt support. Anthropic’s Claude, OpenAI’s Operator, and other browser-agent platforms have no obligation to follow Google Search’s position on llms.txt and have independent reasons to value efficient site orientation during live agent sessions. Explicit public documentation from any of these platforms confirming that they actively parse and act on llms.txt during agent browsing sessions would significantly sharpen the business case for implementation — and would resolve the open adoption question independently of Google’s internal disagreement.

Google reconciling its public guidance. The current contradiction between Search and Chrome teams is publicly documented and visible enough that some form of clarifying communication is likely. Watch for either a joint Google guidance document that explicitly distinguishes search AI optimization from agentic browsing optimization — the framing these two teams would both accept — or for the Search team to quietly update its position if agent-driven sessions become measurable and attributable in Search Console data. Either development would provide the clear operational signal that marketers currently lack.

Content licensing standards incorporating llms.txt as a permissions layer. The file’s potential as a machine-readable permissions instrument — communicating to agents what content may be consumed, summarized, or reproduced and under what conditions — is the most legally significant near-term development in the spec’s evolution. As AI content licensing disputes progress through courts and regulatory bodies across the EU and US, a standardized, machine-readable permissions syntax in llms.txt could become both a legal instrument and an industry norm. The llmstxt.org spec does not currently include formal permissions syntax, but the community is actively discussing this extension on GitHub.

Bottom Line

Google Search is right that llms.txt doesn’t affect your AI Overviews rankings, and Google Lighthouse is right that it matters for agentic browsing readiness — these are both accurate statements about different channels with different mechanics, and the confusion arises only when practitioners treat them as a single unified question. The practical path forward is to stop asking “should I implement llms.txt for SEO” and start asking “have I prepared for agentic browsing as a distinct optimization channel” — and then implement accordingly. Run the Lighthouse Agentic Browsing audits, deploy a well-structured llms.txt, fix your ARIA labeling and CLS issues, and build the two-track AI optimization framework your team needs to navigate an AI web that is fragmenting faster than any single vendor’s guidance can keep up with. The marketers who establish this infrastructure now, while the traffic is still small and the remediation cost is low, will be operating from a position of strength when browser-agent sessions become a material line in their analytics.