--- title: "Sitemap and robots.txt" url: "https://glokki.com/docs/sitemap-and-robots" locale: "en" updated: "2026-10-14T21:07:15.945Z" --- # Sitemap and robots.txt How to serve /sitemap.xml and /robots.txt from your tenant app — using the sitemap the CMS already generates, and a robots.txt that keeps the editor preview route and non-production deployments out of search engines. The CMS builds an XML sitemap of every published, indexable page in your tenant and serves it from the Delivery API. Your app only needs a thin route that fetches it server-side and re-serves it from your own domain, plus a `robots.txt` that points crawlers at it. For the rest of the delivery integration, see the [Integration Guide](/docs/integration). --- ## Table of Contents 1. [The Sitemap Endpoint](#1-the-sitemap-endpoint) 2. [Serving `/sitemap.xml`](#2-serving-sitemapxml) 3. [Serving `/robots.txt`](#3-serving-robotstxt) 4. [Verifying the Setup](#4-verifying-the-setup) 5. [Troubleshooting](#5-troubleshooting) --- ## 1. The Sitemap Endpoint ``` GET {CMS_BASE_URL}/api/v1/delivery/sitemap.xml Authorization: Bearer X-Tenant-Slug: ``` There is no SDK client method for it — call it with `fetch` from server-side code. It uses the same `CMS_BASE_URL`, `CMS_TENANT`, and `CMS_API_KEY` values as the rest of the integration; the key needs the `delivery:read` scope and must never be sent to the browser. ### Query parameters | Param | Required | Description | | ---------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `base_url` | No | The origin used to build every `` URL, e.g. `https://acme.com`. Trailing slashes are stripped. When omitted, the CMS uses your **Frontend URL** (CMS Dashboard → **Settings → Visual Editor → Frontend URL**). If neither is set, the endpoint returns `422 FRONTEND_URL_NOT_CONFIGURED`. | ### What it contains - **Only published pages.** The sitemap is built from each page's published version, so unpublished drafts and unsaved edits never appear. Collection entries that are published as pages are included too. - **No `noindex` pages.** A page whose published robots directive contains `noindex` (the _No Index, …_ options in the page's SEO settings) is left out. - **``** — the date (`YYYY-MM-DD`) the page was last published. - **hreflang alternates.** When pages with the **same slug** exist in several locales, each `` lists every locale as an ``, plus an `x-default` entry pointing at the `en` version. > **Locale URLs follow a fixed scheme.** In the sitemap, `en` pages are at the root (`https://acme.com/about`) and every other locale gets a path prefix (`https://acme.com/de/about`). Your tenant's configured default locale and locale URL strategy are not applied to the sitemap, and translations with different slugs (`/about` vs `/de/ueber-uns`) are not linked as alternates. If your site is single-locale in `en`, or uses exactly this scheme, nothing needs doing. Otherwise, post-process the XML in your route (see [section 2](#2-serving-sitemapxml)) or build the sitemap yourself. ### Response | Header | Value | | --------------- | ----------------------------------------------------------------------------- | | `Content-Type` | `application/xml; charset=utf-8` | | `Cache-Control` | `public, s-maxage=3600, stale-while-revalidate=86400` | | `ETag` | A hash of the XML. Send it back as `If-None-Match` to get `304 Not Modified`. | | `Vary` | `X-Tenant-Slug, Host` | The endpoint counts against your API key's delivery rate limit like every other Delivery API call, and returns `429` when the budget is exhausted — another reason to let a CDN cache your `/sitemap.xml` rather than proxying every crawler hit through to the CMS. ### Why not build it from `pages.list()`? The SDK's `pages.list()` returns lightweight items — slug, title, locale, `publishedAt` — with no robots directive, no locale alternates, and no last-modified date. Building a correct sitemap from it would mean one `pages.getBySlug()` call per page just to find out which pages are `noindex` and which translations belong together. The sitemap endpoint does that in a single query. --- ## 2. Serving `/sitemap.xml` Add a route handler that fetches the CMS sitemap server-side and returns it unchanged: ```ts // app/sitemap.xml/route.ts const CMS_BASE_URL = process.env.CMS_BASE_URL!.replace(/\/+$/, "") // Your public origin, e.g. https://acme.com — the host every will use. const SITE_URL = process.env.NEXT_PUBLIC_SITE_URL! export async function GET(request: Request) { const url = new URL(`${CMS_BASE_URL}/api/v1/delivery/sitemap.xml`) url.searchParams.set("base_url", SITE_URL) const headers: Record = { Authorization: `Bearer ${process.env.CMS_API_KEY}`, "X-Tenant-Slug": process.env.CMS_TENANT!, } const ifNoneMatch = request.headers.get("if-none-match") if (ifNoneMatch) headers["If-None-Match"] = ifNoneMatch let upstream: Response try { upstream = await fetch(url, { headers, cache: "no-store" }) } catch { return unavailable() } if (upstream.status === 304) { return new Response(null, { status: 304, headers: passthrough(upstream) }) } if (!upstream.ok) return unavailable() return new Response(upstream.body, { status: 200, headers: passthrough(upstream), }) } function passthrough(upstream: Response): Record { const out: Record = {} for (const name of ["content-type", "cache-control", "etag"]) { const value = upstream.headers.get(name) if (value) out[name] = value } return out } // A 5xx tells crawlers to come back later instead of treating the sitemap as gone. function unavailable() { return new Response("Sitemap temporarily unavailable", { status: 503, headers: { "Retry-After": "300", "Cache-Control": "no-store" }, }) } ``` Notes: - **Pass `base_url` explicitly.** It ties `` URLs to the origin this deployment actually serves, independent of what is configured in the CMS. Without it, the Frontend URL from **Settings → Visual Editor** is used. - **Caching.** The CMS's `Cache-Control` (`s-maxage=3600`, `stale-while-revalidate=86400`) is passed through, so a CDN in front of your app — Vercel's included — caches the sitemap for an hour and refreshes it in the background. Newly published pages appear within that window. The upstream `fetch` uses `cache: "no-store"` so Next.js does not add a second, independent cache layer on top. - **Errors.** Any upstream failure becomes a `503` with `Retry-After`. Don't turn an upstream `401` or `422` into a `200` with an empty `` — crawlers would read that as "this site has no pages". - **Post-processing.** If you need to adjust the XML (for example to rewrite locale prefixes, see the note in [section 1](#what-it-contains)), read `await upstream.text()`, transform it, and drop the upstream `etag` header since the body no longer matches it. ### Why a route handler and not `app/sitemap.ts`? Next.js's `app/sitemap.ts` convention (`MetadataRoute.Sitemap`) expects you to return an array of entry objects, which Next.js then serialises to XML. The CMS already returns finished XML, so using `sitemap.ts` would mean parsing that XML back into objects only for Next.js to serialise it again — or assembling the data from `pages.list()` and per-page `getBySlug()` calls, as explained [above](#why-not-build-it-from-pageslist). A route handler at `app/sitemap.xml/route.ts` passes the XML through verbatim, including its `ETag` and cache headers. Use one or the other: an `app/sitemap.ts` and an `app/sitemap.xml/route.ts` in the same app both claim `/sitemap.xml`. --- ## 3. Serving `/robots.txt` Use Next.js's `app/robots.ts` convention: ```ts // app/robots.ts import type { MetadataRoute } from "next" const SITE_URL = process.env.NEXT_PUBLIC_SITE_URL! // e.g. https://acme.com // Must match Settings → Visual Editor → Editor Preview Path in the CMS. const EDITOR_PREVIEW_PATH = "/_editor-preview" export default function robots(): MetadataRoute.Robots { // Staging, preview deployments, local dev: keep crawlers out entirely. if (process.env.APP_ENV !== "production") { return { rules: { userAgent: "*", disallow: "/" } } } return { rules: { userAgent: "*", allow: "/", disallow: [EDITOR_PREVIEW_PATH], }, sitemap: `${SITE_URL}/sitemap.xml`, } } ``` What each part does: - **`Disallow: /_editor-preview`** — the route the CMS loads to edit block templates (see [Editor Preview Route](/docs/editor-preview-route)). It renders nothing outside the editor, so there is nothing for a crawler there. The default path is `/_editor-preview`; if you changed it under **CMS Dashboard → Settings → Visual Editor → Editor Preview Path**, use the same value here. - **`Sitemap:`** — an absolute URL to the route from [section 2](#2-serving-sitemapxml). Use the same origin you pass as `base_url`. - **Full block outside production.** `cmsMetadata()` from `@glokki-cms/react` already forces `noindex, nofollow` on every page unless `APP_ENV` is `production`. Keying `robots.txt` off the same variable keeps the two consistent: one setting — `APP_ENV=production` on your production deployment only — opens the site to search engines, and every other environment stays closed. If `APP_ENV` is not set in production, both the meta tags and `robots.txt` will keep the live site out of search results. Do **not** disallow `/cdn/` or `/og/`. Images under `/cdn/` belong in image search, and social-card crawlers such as Twitterbot honour `robots.txt` — blocking `/og/` (see [OG Image Proxy](/docs/asset-cdn-proxy#7-og-image-proxy)) would break link previews. A note on combining the two mechanisms: a crawler that is blocked by `robots.txt` never fetches the page, so it never sees its `noindex` meta tag. For a staging host that has never been indexed, blocking is exactly what you want. If a non-production host **has** already been indexed, temporarily allow crawling so search engines can see the `noindex` and drop the pages, then block it again. Use either `app/robots.ts` or a static `public/robots.txt`, not both — they serve the same URL. The static file cannot switch on `APP_ENV`, so prefer `robots.ts`. --- ## 4. Verifying the Setup 1. **Fetch the sitemap from your site:** ```bash curl -s https://acme.com/sitemap.xml | head -20 ``` Expect ``, a `` element, and one `` with `` and `` per published page. Every `` should start with your `NEXT_PUBLIC_SITE_URL`. 2. **Check a `noindex` page is absent.** Set a page to _No Index, Follow_ in its SEO settings, publish it, and confirm its URL is missing from the sitemap (allow up to an hour if a CDN cached the previous version). 3. **Check caching:** ```bash curl -sI https://acme.com/sitemap.xml | grep -i -E "cache-control|etag" # cache-control: public, s-maxage=3600, stale-while-revalidate=86400 ``` 4. **Fetch `robots.txt` on production and on a non-production deployment:** ```bash curl -s https://acme.com/robots.txt # User-Agent: * # Allow: / # Disallow: /_editor-preview # # Sitemap: https://acme.com/sitemap.xml curl -s https://staging.acme.com/robots.txt # User-Agent: * # Disallow: / ``` 5. **Submit the sitemap** in Google Search Console (and Bing Webmaster Tools) once production looks right. --- ## 5. Troubleshooting ### `/sitemap.xml` returns `503` Your route caught an upstream failure. Call the CMS endpoint directly with the same headers to see the real status: ```bash curl -i -H "Authorization: Bearer $CMS_API_KEY" -H "X-Tenant-Slug: $CMS_TENANT" \ "$CMS_BASE_URL/api/v1/delivery/sitemap.xml?base_url=https://acme.com" ``` - `401` — the API key is missing, invalid, expired, or revoked. - `403` — the API key lacks the `delivery:read` scope. - `422 FRONTEND_URL_NOT_CONFIGURED` — no `base_url` was sent and no Frontend URL is configured. - `429` — the delivery rate limit is exhausted; make sure a CDN is caching your `/sitemap.xml`. ### The sitemap is empty The `` has no `` entries when the tenant has no published pages, or every published page is `noindex`. Drafts are never listed — publish the pages. ### `` URLs use the wrong host Without `base_url`, the CMS uses the Frontend URL from **Settings → Visual Editor**, which may point at a staging or preview host. Pass `base_url` explicitly from your route. ### Locale URLs don't match my routes See the locale note in [What it contains](#what-it-contains): the sitemap uses unprefixed URLs for `en` and `/{locale}/` prefixes for everything else. ### Next.js reports a conflict on `/sitemap.xml` or `/robots.txt` You have two sources for the same URL — `app/sitemap.ts` alongside `app/sitemap.xml/route.ts`, or `app/robots.ts` alongside `public/robots.txt`. Keep one. --- ## Related Guides - [Integration Guide](/docs/integration) — `CMS_BASE_URL` / `CMS_TENANT` / `CMS_API_KEY`, and `cmsMetadata()` for per-page robots and hreflang tags. - [Editor Preview Route](/docs/editor-preview-route) — the route `robots.txt` disallows. - [Asset CDN Proxy](/docs/asset-cdn-proxy) — the `/cdn/` and `/og/` routes that must stay crawlable.