Sitemap and robots.txt

The CMS builds an XML sitemap of every published, indexable page in your tenant and serves it from the Delivery API. Your app only needs a thin route that fetches it server-side and re-serves it from your own domain, plus a robots.txt that points crawlers at it. For the rest of the delivery integration, see the Integration Guide.

1. The Sitemap Endpoint

GET {CMS_BASE_URL}/api/v1/delivery/sitemap.xml
Authorization: Bearer <delivery API key>
X-Tenant-Slug: <your tenant slug>

There is no SDK client method for it — call it with fetch from server-side code. It uses the same CMS_BASE_URL, CMS_TENANT, and CMS_API_KEY values as the rest of the integration; the key needs the delivery:read scope and must never be sent to the browser.

Query parameters

What it contains

• Only published pages. The sitemap is built from each page's published version, so unpublished drafts and unsaved edits never appear. Collection entries that are published as pages are included too.
• No noindex pages. A page whose published robots directive contains noindex (the No Index, … options in the page's SEO settings) is left out.
• <lastmod> — the date (YYYY-MM-DD) the page was last published.
• hreflang alternates. When pages with the same slug exist in several locales, each <url> lists every locale as an <xhtml:link rel="alternate" hreflang="...">, plus an x-default entry pointing at the en version.

Locale URLs follow a fixed scheme. In the sitemap, en pages are at the root (https://acme.com/about) and every other locale gets a path prefix (https://acme.com/de/about). Your tenant's configured default locale and locale URL strategy are not applied to the sitemap, and translations with different slugs (/about vs /de/ueber-uns) are not linked as alternates. If your site is single-locale in en, or uses exactly this scheme, nothing needs doing. Otherwise, post-process the XML in your route (see section 2) or build the sitemap yourself.

Response

The endpoint counts against your API key's delivery rate limit like every other Delivery API call, and returns 429 when the budget is exhausted — another reason to let a CDN cache your /sitemap.xml rather than proxying every crawler hit through to the CMS.

Why not build it from pages.list()?

The SDK's pages.list() returns lightweight items — slug, title, locale, publishedAt — with no robots directive, no locale alternates, and no last-modified date. Building a correct sitemap from it would mean one pages.getBySlug() call per page just to find out which pages are noindex and which translations belong together. The sitemap endpoint does that in a single query.

2. Serving /sitemap.xml

Add a route handler that fetches the CMS sitemap server-side and returns it unchanged:

// app/sitemap.xml/route.ts
const CMS_BASE_URL = process.env.CMS_BASE_URL!.replace(/\/+$/, "")
// Your public origin, e.g. https://acme.com — the host every <loc> will use.
const SITE_URL = process.env.NEXT_PUBLIC_SITE_URL!

export async function GET(request: Request) {
  const url = new URL(`${CMS_BASE_URL}/api/v1/delivery/sitemap.xml`)
  url.searchParams.set("base_url", SITE_URL)

  const headers: Record<string, string> = {
    Authorization: `Bearer ${process.env.CMS_API_KEY}`,
    "X-Tenant-Slug": process.env.CMS_TENANT!,
  }
  const ifNoneMatch = request.headers.get("if-none-match")
  if (ifNoneMatch) headers["If-None-Match"] = ifNoneMatch

  let upstream: Response
  try {
    upstream = await fetch(url, { headers, cache: "no-store" })
  } catch {
    return unavailable()
  }

  if (upstream.status === 304) {
    return new Response(null, { status: 304, headers: passthrough(upstream) })
  }
  if (!upstream.ok) return unavailable()

  return new Response(upstream.body, {
    status: 200,
    headers: passthrough(upstream),
  })
}

function passthrough(upstream: Response): Record<string, string> {
  const out: Record<string, string> = {}
  for (const name of ["content-type", "cache-control", "etag"]) {
    const value = upstream.headers.get(name)
    if (value) out[name] = value
  }
  return out
}

// A 5xx tells crawlers to come back later instead of treating the sitemap as gone.
function unavailable() {
  return new Response("Sitemap temporarily unavailable", {
    status: 503,
    headers: { "Retry-After": "300", "Cache-Control": "no-store" },
  })
}

Notes:

• Pass base_url explicitly. It ties <loc> URLs to the origin this deployment actually serves, independent of what is configured in the CMS. Without it, the Frontend URL from Settings → Visual Editor is used.
• Caching. The CMS's Cache-Control (s-maxage=3600, stale-while-revalidate=86400) is passed through, so a CDN in front of your app — Vercel's included — caches the sitemap for an hour and refreshes it in the background. Newly published pages appear within that window. The upstream fetch uses cache: "no-store" so Next.js does not add a second, independent cache layer on top.
• Errors. Any upstream failure becomes a 503 with Retry-After. Don't turn an upstream 401 or 422 into a 200 with an empty <urlset> — crawlers would read that as "this site has no pages".
• Post-processing. If you need to adjust the XML (for example to rewrite locale prefixes, see the note in section 1), read await upstream.text(), transform it, and drop the upstream etag header since the body no longer matches it.

Why a route handler and not app/sitemap.ts?

Next.js's app/sitemap.ts convention (MetadataRoute.Sitemap) expects you to return an array of entry objects, which Next.js then serialises to XML. The CMS already returns finished XML, so using sitemap.ts would mean parsing that XML back into objects only for Next.js to serialise it again — or assembling the data from pages.list() and per-page getBySlug() calls, as explained above. A route handler at app/sitemap.xml/route.ts passes the XML through verbatim, including its ETag and cache headers.

Use one or the other: an app/sitemap.ts and an app/sitemap.xml/route.ts in the same app both claim /sitemap.xml.

3. Serving /robots.txt

Use Next.js's app/robots.ts convention:

// app/robots.ts
import type { MetadataRoute } from "next"

const SITE_URL = process.env.NEXT_PUBLIC_SITE_URL! // e.g. https://acme.com

// Must match Settings → Visual Editor → Editor Preview Path in the CMS.
const EDITOR_PREVIEW_PATH = "/_editor-preview"

export default function robots(): MetadataRoute.Robots {
  // Staging, preview deployments, local dev: keep crawlers out entirely.
  if (process.env.APP_ENV !== "production") {
    return { rules: { userAgent: "*", disallow: "/" } }
  }

  return {
    rules: {
      userAgent: "*",
      allow: "/",
      disallow: [EDITOR_PREVIEW_PATH],
    },
    sitemap: `${SITE_URL}/sitemap.xml`,
  }
}

What each part does:

• Disallow: /_editor-preview — the route the CMS loads to edit block templates (see Editor Preview Route). It renders nothing outside the editor, so there is nothing for a crawler there. The default path is /_editor-preview; if you changed it under CMS Dashboard → Settings → Visual Editor → Editor Preview Path, use the same value here.
• Sitemap: — an absolute URL to the route from section 2. Use the same origin you pass as base_url.
• Full block outside production. cmsMetadata() from @glokki-cms/react already forces noindex, nofollow on every page unless APP_ENV is production. Keying robots.txt off the same variable keeps the two consistent: one setting — APP_ENV=production on your production deployment only — opens the site to search engines, and every other environment stays closed. If APP_ENV is not set in production, both the meta tags and robots.txt will keep the live site out of search results.

Do not disallow /cdn/ or /og/. Images under /cdn/ belong in image search, and social-card crawlers such as Twitterbot honour robots.txt — blocking /og/ (see OG Image Proxy) would break link previews.

A note on combining the two mechanisms: a crawler that is blocked by robots.txt never fetches the page, so it never sees its noindex meta tag. For a staging host that has never been indexed, blocking is exactly what you want. If a non-production host has already been indexed, temporarily allow crawling so search engines can see the noindex and drop the pages, then block it again.

Use either app/robots.ts or a static public/robots.txt, not both — they serve the same URL. The static file cannot switch on APP_ENV, so prefer robots.ts.

4. Verifying the Setup

1. Fetch the sitemap from your site:

   curl -s https://acme.com/sitemap.xml | head -20
   

   Expect <?xml ...?>, a <urlset ...> element, and one <url> with <loc> and <lastmod> per published page. Every <loc> should start with your NEXT_PUBLIC_SITE_URL.

2. Check a noindex page is absent. Set a page to No Index, Follow in its SEO settings, publish it, and confirm its URL is missing from the sitemap (allow up to an hour if a CDN cached the previous version).

3. Check caching:

   curl -sI https://acme.com/sitemap.xml | grep -i -E "cache-control|etag"
   # cache-control: public, s-maxage=3600, stale-while-revalidate=86400
   

4. Fetch robots.txt on production and on a non-production deployment:

   curl -s https://acme.com/robots.txt
   # User-Agent: *
   # Allow: /
   # Disallow: /_editor-preview
   #
   # Sitemap: https://acme.com/sitemap.xml
   
   curl -s https://staging.acme.com/robots.txt
   # User-Agent: *
   # Disallow: /
   

5. Submit the sitemap in Google Search Console (and Bing Webmaster Tools) once production looks right.

5. Troubleshooting

/sitemap.xml returns 503

Your route caught an upstream failure. Call the CMS endpoint directly with the same headers to see the real status:

curl -i -H "Authorization: Bearer $CMS_API_KEY" -H "X-Tenant-Slug: $CMS_TENANT" \
  "$CMS_BASE_URL/api/v1/delivery/sitemap.xml?base_url=https://acme.com"

• 401 — the API key is missing, invalid, expired, or revoked.
• 403 — the API key lacks the delivery:read scope.
• 422 FRONTEND_URL_NOT_CONFIGURED — no base_url was sent and no Frontend URL is configured.
• 429 — the delivery rate limit is exhausted; make sure a CDN is caching your /sitemap.xml.

The sitemap is empty

The <urlset> has no <url> entries when the tenant has no published pages, or every published page is noindex. Drafts are never listed — publish the pages.

<loc> URLs use the wrong host

Without base_url, the CMS uses the Frontend URL from Settings → Visual Editor, which may point at a staging or preview host. Pass base_url explicitly from your route.

Locale URLs don't match my routes

See the locale note in What it contains: the sitemap uses unprefixed URLs for en and /{locale}/ prefixes for everything else.

Next.js reports a conflict on /sitemap.xml or /robots.txt

You have two sources for the same URL — app/sitemap.ts alongside app/sitemap.xml/route.ts, or app/robots.ts alongside public/robots.txt. Keep one.

Related Guides

• Integration Guide — CMS_BASE_URL / CMS_TENANT / CMS_API_KEY, and cmsMetadata() for per-page robots and hreflang tags.
• Editor Preview Route — the route robots.txt disallows.
• Asset CDN Proxy — the /cdn/ and /og/ routes that must stay crawlable.