Guide

External sources

15 min read 2875 words Updated 11 September 2026

One unified system feeds templates with data from outside the content tree: local files, remote HTTP APIs, SQL databases and CMS databases (WordPress, Drupal, Movable Type) — all behind a single registry, one cache, one secrets rule and one error model. The legacy .Data namespace is untouched. A working project lives in examples/external-sources/.

1External Source → Connector → Parser/Adapter → Normalizer → Cache
2             → Unified Data Registry → Generator → Templates

Configuration

 1external_sources:
 2  enabled: true
 3
 4  cache_dir: .ssg-cache/external-sources
 5  offline: false            # serve HTTP sources from the disk cache only
 6  refresh: false            # ignore fresh cache entries, re-fetch
 7  stale_if_error: true      # fall back to an expired copy when a fetch fails
 8  fail_on_cache_miss: true  # offline + no cached copy = build failure
 9  max_concurrent_sources: 4
10  allowed_hosts: []         # optional HTTP allowlist: "api.example.com", "*.example.com", "127.0.0.1:8787"
11
12  defaults:
13    timeout: 10s
14    cache_ttl: 1h
15    stale_ttl: 24h
16    retries: 2
17    retry_backoff: 500ms
18    max_response_size: 5MB
19    required: true
20    allow_http: false       # applies to every source that does not override it
21    allow_private: false
22
23  sources:
24    local_catalog:
25      type: file
26      path: ./data/catalog.csv
27    products_api:
28      type: http
29      url: https://api.example.com/products
30      format: json
31    products_db:
32      type: sql
33      driver: postgres
34      dsn: "$PRODUCT_DB_DSN"
35      queries:
36        products:
37          sql: SELECT id, name, slug FROM products WHERE published = true
38    wordpress:
39      type: cms
40      adapter: wordpress
41      driver: mysql
42      dsn: "$WORDPRESS_DSN"

Source names must match [a-z][a-z0-9_-]*. Sources load in deterministic (name-sorted) order, up to max_concurrent_sources at a time. A required source's failure aborts the build; an optional one (required: false) warns and is skipped. Every failure names the source, its type and the stage (config, read, fetch, parse, transform, connect, query, import) — and never contains credentials.

Secrets come only from the environment. Values written as "$NAME" resolve from the environment at build time; literal secrets in dsn, auth and jwt-style fields are rejected outright, and a referenced-but-unset variable fails the build naming the variable (never a value).

Environment variables in values

url, headers and query expand environment references inline, so one config serves every environment instead of being generated per environment:

1accommodations:
2  type: http
3  url: "$MY_API_BASE/api/accommodations"     # or "${MY_API_BASE}/api/..."
4  headers:
5    Authorization: "Bearer $API_TOKEN"

1MY_API_BASE=https://api.example.com ssg …          # production
2MY_API_BASE=http://127.0.0.1:8787   ssg … --wrangler-dir=api   # local Worker

Rules:

• $NAME and ${NAME} expand; $$ is a literal $. A $ not followed by a variable name ($5, a trailing $) stays literal.
• A variable that is unset or empty counts as missing.
• Missing variable in a required source (the default) fails the build, naming the variable. In an optional source (required: false, or defaults.required: false) the source is skipped with a warning — so a shared config can carry an env-driven source nobody else has to set up.
• dsn and auth secret fields keep the stricter whole-value form (dsn: "$PRODUCT_DB_DSN"); a literal there is still rejected.

Local files (type: file)

Formats: YAML, JSON, TOML, CSV, XML — inferred from the extension or forced with format:. Files are size-capped by max_response_size.

1rates:
2  type: file
3  path: ./data/rates.csv
4  csv:
5    header: true        # rows become {column: value} maps (default)
6    delimiter: ","

XML maps to nested template-friendly maps: attributes become plain keys, repeated elements collect into lists, text-only elements collapse to strings and mixed content keeps its text under #text.

format: feed — read any syndication feed

SSG could always fetch a feed, but not understand one: Atom puts its entries at .feed.entry, RSS at .rss.channel.item and JSON Feed at .items, with the title, link and date in different places and dates in different encodings. A template had to know which format was on the other end, and swapping a source from RSS to Atom broke it even though the content was identical.

format: feed accepts Atom 1.0, RSS 2.0 or JSON Feed 1.1 and returns one shape. The format is detected from the payload, not the declaration — a .xml URL may be either, and a redirect can change what arrives.

1external_sources:
2  sources:
3    mddb:
4      type: http
5      url: https://mddb.tradik.com/feed.xml
6      format: feed

Dates being real timestamps is what lets items from different feeds sort against each other, which is what makes aggregating several sources possible at all — see feeds: in CONFIGURATION.

format: changelog — a CHANGELOG.md as structured data

A CHANGELOG.md following the Keep a Changelog convention is already the canonical record of what shipped. format: changelog parses it into data, so a "What's New" panel reads from the same file you edit at release time instead of a pre-build script or hand-maintained HTML that goes stale. It must be set explicitly — a .md extension infers nothing.

1external_sources:
2  sources:                 # sources live under `sources:`, not directly here
3    changelog:
4      type: file
5      path: CHANGELOG.md
6      format: changelog

The parsed shape:

Each entry is split so a template can wrap the parts in its own markup:

<h3>What's New in v{{ .ExternalData.changelog.latest.version }}</h3>
<ul>
  {{ range first 6 .ExternalData.changelog.latest.sections.added }}
    <li><strong>{{ .title | safeHTML }}</strong> — {{ .html | safeHTML }}</li>
  {{ end }}
</ul>

Headings parse in both ## [1.8.16] - 2026-08-02 and ## 1.8.16 - 2026-08-02 form, - and * bullets are equivalent, and a bullet wrapped over several lines is joined back into one entry.

Remote HTTP (type: http)

 1products_api:
 2  type: http
 3  url: https://api.example.com/products
 4  format: json          # json | csv | xml (yaml/toml also work)
 5  headers:
 6    Accept: application/json
 7  query:
 8    page: "1"
 9  auth:
10    type: bearer        # bearer | basic | header
11    token: "$API_TOKEN"
12  timeout: 10s
13  cache_ttl: 1h
14  stale_ttl: 24h
15  retries: 2
16  retry_backoff: 500ms

Security, always on:

• HTTPS required; plain http:// needs allow_http: true.
• localhost and private/link-local IPs are refused at dial time, which also defeats DNS rebinding — opt out with allow_private: true for self-hosted APIs.
• allow_http and allow_private can be set per source or once under external_sources.defaults; a source may still override either.
• Optional global allowed_hosts allowlist (exact or *.wildcard). An entry without a port matches the host on any port; an entry with a port (127.0.0.1:8787, api.example.com:443) only matches that port, so a local-dev allowance does not widen to every port on the host.
• Redirects are capped at 5 and every hop is re-validated.
• Responses are size-capped; clearly conflicting Content-Types are rejected.
• Error messages carry the URL without its query string.

Pagination (GO-062)

By default an HTTP source fetches exactly one response. Paginated APIs (WordPress REST returns 10 posts per page!) can opt in per source:

 1posts_api:
 2  type: http
 3  url: https://api.example.com/posts
 4  format: json            # pagination aggregates pages into one JSON array
 5  pagination:
 6    mode: page            # page | link
 7    param: page           # query parameter for mode=page (default: page)
 8    start_page: 1         # first page number (default: 1)
 9    per_page: 100         # page-size value; sent only when > 0
10    per_page_param: per_page  # page-size parameter name (default: per_page)
11    max_pages: 10         # hard guard, default 10, max 1000

mode: page increments the query parameter from start_page; mode: link follows the Link: rel="next" response header (RFC 8288, relative targets resolved). Fetching stops on an empty (or non-array) JSON page, a missing next link, or max_pages — hitting the cap prints a warning so truncation is never silent. Pages are aggregated into a single JSON array before transforms; each page request reuses the standard auth/retry/size-cap machinery, and the aggregated result is what lands in the disk cache.

Retries with linear backoff cover network errors, 429 and 5xx. Successful payloads land in the shared disk cache (<hash>.body + <hash>.meta.json, sha256-verified; corrupted entries are evicted). Within cache_ttl the cache is served without touching the network; after that, a failed refetch can serve the stale copy for stale_ttl (stale_if_error).

CLI: --offline (cache only), --refresh-external-sources (ignore fresh cache), --external-source=NAME (narrow the refresh), --clear-external-cache.

Payload CMS (GO-068)

Payload's REST API is a plain JSON API — GET /api/{collection}?limit=&depth= returns { "docs": [...], "totalDocs": N, ... }. The http connector already covers it; no dedicated adapter is needed. Fetch one page with a high limit and unwrap the docs array with transform.select:

 1external_sources:
 2  enabled: true
 3  sources:
 4    payload_posts:
 5      type: http
 6      url: https://cms.example.com/api/posts
 7      format: json
 8      headers:
 9        Authorization: "$PAYLOAD_API_KEY"   # resolved from the environment
10      query:
11        limit: "1000"          # one request; raise to cover the collection
12        depth: "2"             # expand relationships (authors, media)
13      transform:
14        select: docs           # Payload wraps results in { docs: [...] }

The extracted array lands in .ExternalData.payload_posts for templates. To merge the documents into the site as native pages instead, add mode: content and a content_map saying which document field is the title and which is the body — see Records as pages below. Keep the API key in the environment (PAYLOAD_API_KEY), never in the committed config.

1      mode: content
2      content_map:
3        title: title
4        slug: slug
5        content: content
6        date: publishedAt
7        type: "=post"

Note. The generic pagination: block is not used here: it aggregates pages that are each a bare JSON array, whereas Payload wraps every page in a { docs: [...] } object. Raise limit to fetch the collection in one request instead. A collection larger than a single reasonable limit is the one case that would justify a dedicated Payload preset.

Records as pages

A source's records are data a template can iterate. With a content_map they become pages instead — one page per record, with their own URLs, taxonomy archives, sitemap entries, feeds and .md/.json outputs, exactly like a page written by hand.

The build could already do this for a CMS database. Every other source type accepted mode: content in its configuration and then ignored it, because the one thing a tool cannot guess was missing: which field of a record is the title, and which is the body. content_map says so.

 1external_sources:
 2  enabled: true
 3  sources:
 4    products:
 5      type: file
 6      path: data/products.csv
 7      format: csv
 8      mode: content
 9      content_map:
10        title: name              # a field of the record
11        slug: "product-{{.sku}}" # a template over the record
12        content: description
13        date: updated_at
14        type: "=product"         # a constant: every record gets this
15        taxonomies:
16          tag: categories        # a list field, or "a,b" in one CSV column
17      content_format: markdown   # markdown (default) | html | text
18      content_errors: warn       # warn (default) | strict
19      max_rows: 10000            # the record cap

Three CSV rows become three pages at /product-w-1/, /product-w-2/ and /product-w-3/, each rendered through the product type's template if the theme has one and the post template otherwise.

How a mapping value is read

The leading = is the one piece of syntax worth remembering. Writing type: product looks right and means "the record's product field", which no record has; the build says so and points at "=product".

The fields you can map

title (required), slug, content, excerpt, description, date, modified, type, status, link, image, lang, and taxonomies as a map of taxonomy name to record field.

A missing slug is derived from the title. A missing type is post; page routes the record onto the page pipeline. Dates are read as ISO-8601, YYYY-MM-DD, DD/MM/YYYY, RFC 1123 or a Unix timestamp.

What stops the build, and what only warns

• Two records mapping to the same slug stops it, always. Two pages at one URL means one of them is silently lost, and no policy makes that acceptable. The error suggests the fix: a slug template with a field that is unique.
• More records than max_rows stops it, so a source that grew by a thousand does not quietly become a thousand pages.
• A record with no title is skipped with a warning, and a field the mapping names but the record does not have is warned about and left empty. content_errors: strict promotes both to build failures.

content_format

markdown (the default) and html both pass the body to the same renderer every page body goes through, which renders Markdown and passes HTML through. text is the one that changes something: it escapes the characters that turn prose into markup, so a product description with a 50% *off* in it arrives as its author wrote it rather than half-italic.

Still data, too

A mapped source keeps its .ExternalData.<name> namespace. The records are pages and data, so a template that already iterated them still can.

SQL (type: sql)

Drivers: mysql, mariadb, postgres, sqlite (pure Go, no cgo). Queries live only in configuration — never in templates — and are statically validated: a single statement, SELECT (or WITH … SELECT) only, no piggybacked statements. Each query runs under the source timeout with a hard max_rows cap (default 10000); exceeding it is an error, not a silent truncation. DSNs are scrubbed from driver errors.

 1inventory:
 2  type: sql
 3  driver: mysql
 4  dsn: "$INVENTORY_DSN"
 5  queries:
 6    products:
 7      sql: |
 8        SELECT id, name, slug, price
 9        FROM products
10        WHERE published = true
11      max_rows: 10000

{{ range .ExternalData.inventory.products }}{{ .name }}: {{ .price }}{{ end }}

Operational rules: use a dedicated read-only database user, and TLS for remote connections (configure both in the DSN).

CMS adapters (type: cms)

1wordpress:
2  type: cms
3  adapter: wordpress      # wordpress | drupal | movable_type
4  driver: mysql
5  dsn: "$WORDPRESS_DSN"
6  mode: content           # content (default) | data

mode: content merges the import into the site before URL, translation, taxonomy and collision processing — imported posts render, paginate, join feeds, sitemaps and archives exactly like native content. Imported authors merge without overwriting IDs the local metadata already defines. mode: data only exposes the import under .ExternalData.<name> (pages/posts/authors/ taxonomies/media/metadata maps) — the data view is available in both modes.

WordPress

wp_posts, wp_users, wp_terms/wp_term_taxonomy/wp_term_relationships, wp_postmeta and attachments. post and custom post types render as posts; page maps to pages. category/post_tag feed the legacy fields; other taxonomies land in the page's taxonomies map, so dynamic taxonomies pick them up. User-facing custom fields (keys not starting with _) land in .Extra.

1wordpress:
2  table_prefix: wp_
3  post_types: [post, page, guide]
4  statuses: [publish]
5  include_media: true
6  include_custom_fields: true
7  include_taxonomies: true

Drupal (8–11)

node_field_data, node__body, users_field_data, taxonomy_term_field_data/taxonomy_index (vocabularies → taxonomies map) and path_alias — aliases are preserved as explicit links, so Drupal URLs survive the migration. With include_fields: true, dynamic node__field_* tables are discovered per engine and land in .Extra. Drupal 7 uses a different schema and is deferred.

1drupal:
2  version: 10
3  bundles: [article, page]
4  published_only: true
5  include_fields: true

Movable Type

mt_entry (released entries and pages), mt_author, mt_category/mt_placement, mt_tag/mt_objecttag, mt_asset and — opt-in — visible mt_comment rows (GO-058).

1movable_type:
2  include_entries: true
3  include_pages: true
4  include_assets: true
5  include_comments: true   # visible comments → the page's .Extra["comments"]

With include_comments: true each imported entry carries its approved (comment_visible = 1) comments as a list of maps (author, email, url, created_on, text) under .Extra["comments"], ready for templates. Unapproved and spam comments are never imported. Default false — identical behaviour to previous releases.

Template API

{{ .ExternalData.products }}                     — parsed data per source
{{ .ExternalDataMeta.products.FetchedAt }}       — metadata (index and page contexts)
{{ getExternal "products" }}                     — helper, works in every context
{{ getExternalMeta "products" }}                 — Metadata struct

Metadata fields: SourceType, Identifier (always credential-free), FetchedAt, FromCache, Stale, Checksum, RecordCount, ContentType.

Transformations

1transform:
2  select: data.items    # dot path into the parsed structure

select unwraps API envelopes before the data reaches templates. There is deliberately no scripting, eval or embedded query runtime.

Migration notes

• .Data (local data/ files) is unchanged; the new system is additive.
• MDDB remains a separate content source; it may implement the same connector interface later.
• Builds stay deterministic offline: --offline + a warmed cache produces byte-identical output.

Troubleshooting

• failed at fetch … blocked address — the host resolves to a private IP; set allow_private: true for self-hosted APIs.
• offline mode and no cached copy — run once online, or set fail_on_cache_miss: false to downgrade the miss to a warning.
• must reference an environment variable — secrets belong in the environment, not the config file.
• result exceeds max_rows — raise the query's max_rows or narrow the SQL.

Deferred (phase 7+)

• Direct-URL template helpers (getJSON/getCSV/getXML).
• Adapters: Ghost, Strapi, Contentful, Sanity, Notion, Airtable, Google Sheets, GitHub, GitLab; Drupal 7.
• watch: true rebuilds on file-source changes.
• Example CMS projects with seed scripts.