How to make your website discoverable by AI: llms.txt, Markdown and JSON-LD

More and more people ask an assistant instead of a search engine. The assistant fetches a few pages, reads them and answers — often without the person ever seeing your site. Whether your page ends up in that answer depends on how easy it is to read for a program that has a limited budget of tokens and no patience for cookie banners.

There is no single switch for this, and nobody can promise you a place in an AI answer. But a few things make a site much easier for agents to read, and none of them hurt the site for people. Here is what glokki.com does.

1. A site map for LLMs: llms.txt

llms.txt is a proposed convention: a Markdown file at the root of the site with its name, a one-line summary and a list of links to the pages that matter, each with a short description. It tells an agent where to look without crawling the whole site.

Ours is generated from published pages on every request, so it can't go stale: the docs grouped like the docs sidebar, blog posts newest first, then the other pages. It is about 4 KB. Pages marked noindex are left out.

There is also /llms-full.txt, every page's content in one file (about 420 KB for this site), for agents that would rather read everything at once.

2. Markdown versions of every page

HTML is a poor format to hand to a model. A typical page is mostly navigation, scripts and class names, and the content is spread across components. So every page on glokki.com is also available as Markdown, two ways:

# append .md to any URL
curl https://glokki.com/docs/mcp-server.md

# or ask for Markdown on the normal URL
curl -H "Accept: text/markdown" https://glokki.com/docs

The second form answers with Content-Type: text/markdown, Vary: Accept so caches keep the HTML and Markdown versions apart, and a Link header pointing at the HTML page as canonical.

The Markdown is not scraped from the HTML. Each block on the site declares how it renders to Markdown, next to its React component — a FAQ block becomes questions as headings with their answers, a feature grid becomes a heading per card, and links that only jump within the HTML page (#faq) are dropped. That keeps the Markdown short and keeps it in step with the page.

3. Structured data with JSON-LD

JSON-LD is a <script type="application/ld+json"> block that states facts about the page in a vocabulary search engines and assistants understand: this is an article, published on this date, by this author.

In glokki, any page can carry JSON-LD from its SEO settings, and blocks can emit their own. We use both: the home page has an Organization entry (name, URL, logo, a one-line description) set in its SEO settings in the CMS, and blog posts emit BlogPosting with the headline, dates, author and image.

What we don't do yet: the docs have no structured data. TechArticle on each guide is the obvious next step.

4. Honest crawl rules and canonical URLs

The basics still apply, and agents rely on them as much as search engines:

• /sitemap.xml lists every published, indexable page, with the publish date and language alternates.
• /robots.txt keeps crawlers out of the editor preview route and API routes, and keeps preview deployments out of the index entirely.
• Every Markdown response points back to its HTML page as canonical, so the two versions don't compete.

One trap we found while building this site: the SDK helper that builds page metadata sets noindex on everything unless APP_ENV=production, and our docs didn't say so. A site set up by the book would have been hidden from search engines. If you do nothing else, check your production pages for a stray noindex.

What this does not do

None of this guarantees that an assistant will cite you. It removes reasons not to: a page that is easy to find, fast to read and clear about what it is has a better chance than one that isn't. Content still matters most — answer the question the person asked, near the top of the page.

Try it on your site

All four come with glokki: llms.txt, Markdown for every page, JSON-LD in the SEO settings, and the sitemap. The Markdown for agents guide shows the setup, and the sitemap and robots.txt guide covers the crawl rules.

Create a free account and see how your site reads to an agent.