AI-index for a company: why does a site need a page for AI agents

Previously, a corporate website had two recipients: a person and a search robot. Now the third - agent who reads the page to answer the user's question. He doesn’t scroll through the results, doesn’t click ten links, and almost never comes back: he picks up a few fragments, collects the answer and indicates the source - or does not indicate it.

The difference is fundamental. You are competing with the search engine for position in the list. To the agent - so that your text generally falls into the context of the response and is recognized as reliable. Optimizing for the first does not guarantee the second: a page can rank perfectly and at the same time be unsuitable for citing - because the fact is spread out over three paragraphs, is not directly named anywhere and does not have a date.

The agent is designed differently. In simple terms, it does three things: searches (often through a regular search API), fetches several entire pages, and puts their fragments into a limited context model. This has practical consequences.

• Scripts are usually not executed. If the content is rendered on the client, there is no page for the agent.
• Context is limited. The response will not include the entire page, but several fragments. The winner is a text in which the fact is placed in one paragraph along with its context - and not “as we already wrote above.”
• There is no second chance. The agent will not click “more details”. Key information should be on the same page where he came.
• Attribution requires certainty. To link to you, the agent needs to clearly understand who the author is, what kind of organization it is and when it was written. The untitled, undated text is cited less frequently.
• Controversy is costly. If the site has two different answers to one question, the agent will choose a random one - or someone else's source.

Files in the root

• robots.txt - not only prohibitions. It is important not to block agents by inertia if you want to get into their responses. A separate policy issue: the decision to “let it in or not” should be a conscious one, not an inherited one.
• sitemap.xml - complete, with last modified dates, without junk addresses. For a multilingual site - with all language versions.
• llms.txt - community-proposed convention: a short machine-readable overview of the site with links to key sections. It is not an official standard, but it is cheap and solves the problem of “giving the agent a card.” Advanced option llms-full.txt contains a summary of the entire content.

Page layout

• Structured Data (schema.org in JSON-LD format): Organization with details, Article with author and date, BreadcrumbList, FAQPage for questions and answers. This is the only way to tell a machine a fact without parsing the text.
• Canonical addresses and correct hreflang for language versions - otherwise the duplicates will blur the signal.
• Hierarchy of headings. One h1, meaningful h2/h3. Fragments are cut from them and the table of contents of the answer is built on them.
• Tables instead of paragraphs where the data is tabular: the fact is extracted from the table unambiguously.
• Meaningful addresses and page titles - they often become a caption to a quote.

Infrastructure

• Server rendering or static generation for all relevant content.
• Fast and stable response: the agent does not wait long and does not try again.
• The absence of mandatory banners and modal windows on top of the content - some parsers rely on them.
• Honest response codes: 404 for absentee, 301 for moved. A “soft” 404 with a stub page poisons the index.

Availability

• Content is visible when JavaScript is disabled.
• robots.txt does not block the agents you want to see, and the decision about this is made consciously.
• sitemap.xml complete, with modification dates, no junk URLs.
• Eat llms.txt with a map of key sections.
• The server response is fast and stable, without mandatory modal windows on top of the content.

Marking

• Each page has JSON-LD: organization, page type, author, date.
• Canonical addresses are specified and hreflang for language versions.
• The heading hierarchy is maintained, h1 one.
• Tabular data is formatted as tables rather than lists in the text.

Content

• Each section begins with a direct answer.
• Paragraphs are self-contained and understandable outside the context of the page.
• Facts have dates, statements have reasons.
• Details, licenses and supporting documents are available on a separate page without registration.
• No two pages answer the same question differently.

Observation

• Agent calls are highlighted in the logs, their share and dynamics are known.
• There is a regular check: what popular assistants answer to key questions about your company.
• After publication, forced re-indexing of the changed addresses is launched.

Conventional metrics work poorly here: the agent often does not bring traffic at all, but forms an impression for the user. What is measurable in practice:

• Share of agent calls in logs — by User-Agent. Grows with their penetration; a sharp dip usually means you accidentally blocked them.
• Regular survey of assistants. A list of 20–30 questions about the company and its topics, monthly verification of answers: what they say, whether they refer to you, whether there is outdated information.
• Index completeness — the proportion of pages from the site map that are included in the search index.
• Time until indexing new publication.
• Share of pages with correct structured markup — checked automatically when building the site.