# External sources One unified system feeds templates with data from outside the content tree: local files, remote HTTP APIs, SQL databases and CMS databases (WordPress, Drupal, Movable Type) — all behind a single registry, one cache, one secrets rule and one error model. The legacy `.Data` namespace is untouched. A working project lives in [`examples/external-sources/`](../examples/external-sources/). ```text External Source → Connector → Parser/Adapter → Normalizer → Cache → Unified Data Registry → Generator → Templates ``` ## Configuration ```yaml external_sources: enabled: true cache_dir: .ssg-cache/external-sources offline: false # serve HTTP sources from the disk cache only refresh: false # ignore fresh cache entries, re-fetch stale_if_error: true # fall back to an expired copy when a fetch fails fail_on_cache_miss: true # offline + no cached copy = build failure max_concurrent_sources: 4 allowed_hosts: [] # optional HTTP allowlist: "api.example.com", "*.example.com", "127.0.0.1:8787" defaults: timeout: 10s cache_ttl: 1h stale_ttl: 24h retries: 2 retry_backoff: 500ms max_response_size: 5MB required: true allow_http: false # applies to every source that does not override it allow_private: false sources: local_catalog: type: file path: ./data/catalog.csv products_api: type: http url: https://api.example.com/products format: json products_db: type: sql driver: postgres dsn: "$PRODUCT_DB_DSN" queries: products: sql: SELECT id, name, slug FROM products WHERE published = true wordpress: type: cms adapter: wordpress driver: mysql dsn: "$WORDPRESS_DSN" ``` Source names must match `[a-z][a-z0-9_-]*`. Sources load in deterministic (name-sorted) order, up to `max_concurrent_sources` at a time. A `required` source's failure aborts the build; an optional one (`required: false`) warns and is skipped. Every failure names the source, its type and the stage (`config`, `read`, `fetch`, `parse`, `transform`, `connect`, `query`, `import`) — and never contains credentials. **Secrets come only from the environment.** Values written as `"$NAME"` resolve from the environment at build time; literal secrets in `dsn`, `auth` and `jwt`-style fields are rejected outright, and a referenced-but-unset variable fails the build naming the variable (never a value). ### Environment variables in values `url`, `headers` and `query` expand environment references **inline**, so one config serves every environment instead of being generated per environment: ```yaml accommodations: type: http url: "$MY_API_BASE/api/accommodations" # or "${MY_API_BASE}/api/..." headers: Authorization: "Bearer $API_TOKEN" ``` ```bash MY_API_BASE=https://api.example.com ssg … # production MY_API_BASE=http://127.0.0.1:8787 ssg … --wrangler-dir=api # local Worker ``` Rules: - `$NAME` and `${NAME}` expand; `$$` is a literal `$`. A `$` not followed by a variable name (`$5`, a trailing `$`) stays literal. - A variable that is unset **or empty** counts as missing. - Missing variable in a **required** source (the default) fails the build, naming the variable. In an optional source (`required: false`, or `defaults.required: false`) the source is **skipped with a warning** — so a shared config can carry an env-driven source nobody else has to set up. - `dsn` and `auth` secret fields keep the stricter whole-value form (`dsn: "$PRODUCT_DB_DSN"`); a literal there is still rejected. ## Local files (`type: file`) Formats: YAML, JSON, TOML, CSV, XML — inferred from the extension or forced with `format:`. Files are size-capped by `max_response_size`. ```yaml rates: type: file path: ./data/rates.csv csv: header: true # rows become {column: value} maps (default) delimiter: "," ``` XML maps to nested template-friendly maps: attributes become plain keys, repeated elements collect into lists, text-only elements collapse to strings and mixed content keeps its text under `#text`. ### `format: feed` — read any syndication feed SSG could always *fetch* a feed, but not *understand* one: Atom puts its entries at `.feed.entry`, RSS at `.rss.channel.item` and JSON Feed at `.items`, with the title, link and date in different places and dates in different encodings. A template had to know which format was on the other end, and swapping a source from RSS to Atom broke it even though the content was identical. `format: feed` accepts **Atom 1.0, RSS 2.0 or JSON Feed 1.1** and returns one shape. The format is detected from the payload, not the declaration — a `.xml` URL may be either, and a redirect can change what arrives. ```yaml external_sources: sources: mddb: type: http url: https://mddb.tradik.com/feed.xml format: feed ``` | Path | Contents | |---|---| | `.title`, `.home_page_url` | Feed metadata | | `.items` | Every entry, in feed order | | `.title`, `.url`, `.id` | Identity | | `.summary`, `.content_html` | Body | | `.published`, `.updated` | **Parsed `time.Time`** — the date helpers and sorting work on them | | `.tags`, `.author` | Categories and byline | Dates being real timestamps is what lets items from different feeds sort against each other, which is what makes aggregating several sources possible at all — see `feeds:` in [CONFIGURATION.md](CONFIGURATION.md). ### `format: changelog` — a CHANGELOG.md as structured data A `CHANGELOG.md` following the [Keep a Changelog](https://keepachangelog.com) convention is already the canonical record of what shipped. `format: changelog` parses it into data, so a "What's New" panel reads from the same file you edit at release time instead of a pre-build script or hand-maintained HTML that goes stale. It must be set explicitly — a `.md` extension infers nothing. ```yaml external_sources: sources: # sources live under `sources:`, not directly here changelog: type: file path: CHANGELOG.md format: changelog ``` The parsed shape: | Path | Contents | |---|---| | `.versions` | Every version block, in document order | | `.latest` | The first **released** version (skips `Unreleased`) | | `.unreleased` | The `Unreleased` block, when the file has one | | `.version` / `.date` / `.released` | `"1.8.16"`, `"2026-08-02"`, `true` | | `.sections` | Lowercased `###` headings ⇒ `added`, `changed`, `fixed`, … | | `.entries` | Every bullet of that version, regardless of section | Each entry is split so a template can wrap the parts in its own markup: | Field | Contents | |---|---| | `title` | The leading `**bold**` run, rendered inline (`""` when absent) | | `html` | The rest of the entry, rendered inline | | `full` | The whole entry rendered inline, when you want it in one piece | | `marker` | The leading emoji, when the entry opens with one | | `text` | The whole entry as raw Markdown | ```gotemplate

What's New in v{{ .ExternalData.changelog.latest.version }}

    {{ range first 6 .ExternalData.changelog.latest.sections.added }}
  • {{ .title | safeHTML }} — {{ .html | safeHTML }}
    • {{ end }}
``` Headings parse in both `## [1.8.16] - 2026-08-02` and `## 1.8.16 - 2026-08-02` form, `-` and `*` bullets are equivalent, and a bullet wrapped over several lines is joined back into one entry. ## Remote HTTP (`type: http`) ```yaml products_api: type: http url: https://api.example.com/products format: json # json | csv | xml (yaml/toml also work) headers: Accept: application/json query: page: "1" auth: type: bearer # bearer | basic | header token: "$API_TOKEN" timeout: 10s cache_ttl: 1h stale_ttl: 24h retries: 2 retry_backoff: 500ms ``` Security, always on: - HTTPS required; plain `http://` needs `allow_http: true`. - localhost and private/link-local IPs are refused **at dial time**, which also defeats DNS rebinding — opt out with `allow_private: true` for self-hosted APIs. - `allow_http` and `allow_private` can be set per source **or** once under `external_sources.defaults`; a source may still override either. - Optional global `allowed_hosts` allowlist (exact or `*.wildcard`). An entry without a port matches the host on any port; an entry **with** a port (`127.0.0.1:8787`, `api.example.com:443`) only matches that port, so a local-dev allowance does not widen to every port on the host. - Redirects are capped at 5 and every hop is re-validated. - Responses are size-capped; clearly conflicting `Content-Type`s are rejected. - Error messages carry the URL without its query string. ### Pagination (GO-062) By default an HTTP source fetches exactly one response. Paginated APIs (WordPress REST returns 10 posts per page!) can opt in per source: ```yaml posts_api: type: http url: https://api.example.com/posts format: json # pagination aggregates pages into one JSON array pagination: mode: page # page | link param: page # query parameter for mode=page (default: page) start_page: 1 # first page number (default: 1) per_page: 100 # page-size value; sent only when > 0 per_page_param: per_page # page-size parameter name (default: per_page) max_pages: 10 # hard guard, default 10, max 1000 ``` `mode: page` increments the query parameter from `start_page`; `mode: link` follows the `Link: rel="next"` response header (RFC 8288, relative targets resolved). Fetching stops on an empty (or non-array) JSON page, a missing next link, or `max_pages` — hitting the cap prints a warning so truncation is never silent. Pages are aggregated into a single JSON array before transforms; each page request reuses the standard auth/retry/size-cap machinery, and the aggregated result is what lands in the disk cache. Retries with linear backoff cover network errors, 429 and 5xx. Successful payloads land in the shared disk cache (`.body` + `.meta.json`, sha256-verified; corrupted entries are evicted). Within `cache_ttl` the cache is served without touching the network; after that, a failed refetch can serve the stale copy for `stale_ttl` (`stale_if_error`). CLI: `--offline` (cache only), `--refresh-external-sources` (ignore fresh cache), `--external-source=NAME` (narrow the refresh), `--clear-external-cache`. ### Payload CMS (GO-068) Payload's REST API is a plain JSON API — `GET /api/{collection}?limit=&depth=` returns `{ "docs": [...], "totalDocs": N, ... }`. The `http` connector already covers it; no dedicated adapter is needed. Fetch one page with a high `limit` and unwrap the `docs` array with `transform.select`: ```yaml external_sources: enabled: true sources: payload_posts: type: http url: https://cms.example.com/api/posts format: json headers: Authorization: "$PAYLOAD_API_KEY" # resolved from the environment query: limit: "1000" # one request; raise to cover the collection depth: "2" # expand relationships (authors, media) transform: select: docs # Payload wraps results in { docs: [...] } ``` The extracted array lands in `.ExternalData.payload_posts` for templates. To merge the documents into the site as native pages instead, add `mode: content` **and a `content_map`** saying which document field is the title and which is the body — see [Records as pages](#records-as-pages) below. Keep the API key in the environment (`PAYLOAD_API_KEY`), never in the committed config. ```yaml mode: content content_map: title: title slug: slug content: content date: publishedAt type: "=post" ``` > **Note.** The generic `pagination:` block is not used here: it aggregates > pages that are each a bare JSON array, whereas Payload wraps every page in a > `{ docs: [...] }` object. Raise `limit` to fetch the collection in one > request instead. A collection larger than a single reasonable `limit` is the > one case that would justify a dedicated Payload preset. ## Records as pages A source's records are data a template can iterate. With a `content_map` they become **pages** instead — one page per record, with their own URLs, taxonomy archives, sitemap entries, feeds and `.md`/`.json` outputs, exactly like a page written by hand. The build could already do this for a CMS database. Every other source type accepted `mode: content` in its configuration and then ignored it, because the one thing a tool cannot guess was missing: which field of a record is the title, and which is the body. `content_map` says so. ```yaml external_sources: enabled: true sources: products: type: file path: data/products.csv format: csv mode: content content_map: title: name # a field of the record slug: "product-{{.sku}}" # a template over the record content: description date: updated_at type: "=product" # a constant: every record gets this taxonomies: tag: categories # a list field, or "a,b" in one CSV column content_format: markdown # markdown (default) | html | text content_errors: warn # warn (default) | strict max_rows: 10000 # the record cap ``` Three CSV rows become three pages at `/product-w-1/`, `/product-w-2/` and `/product-w-3/`, each rendered through the `product` type's template if the theme has one and the post template otherwise. ### How a mapping value is read | Written as | Means | |---|---| | `name` | the record's `name` field | | `attributes.title` | a dotted path into a nested record | | `=product` | a **constant** — every record gets this value | | `product-{{.sku}}` | a **template** over the record, for a value built from several fields | The leading `=` is the one piece of syntax worth remembering. Writing `type: product` looks right and means "the record's `product` field", which no record has; the build says so and points at `"=product"`. ### The fields you can map `title` (required), `slug`, `content`, `excerpt`, `description`, `date`, `modified`, `type`, `status`, `link`, `image`, `lang`, and `taxonomies` as a map of taxonomy name to record field. A missing `slug` is derived from the title. A missing `type` is `post`; `page` routes the record onto the page pipeline. Dates are read as ISO-8601, `YYYY-MM-DD`, `DD/MM/YYYY`, RFC 1123 or a Unix timestamp. ### What stops the build, and what only warns - **Two records mapping to the same slug** stops it, always. Two pages at one URL means one of them is silently lost, and no policy makes that acceptable. The error suggests the fix: a slug template with a field that is unique. - **More records than `max_rows`** stops it, so a source that grew by a thousand does not quietly become a thousand pages. - **A record with no title** is skipped with a warning, and a **field the mapping names but the record does not have** is warned about and left empty. `content_errors: strict` promotes both to build failures. ### `content_format` `markdown` (the default) and `html` both pass the body to the same renderer every page body goes through, which renders Markdown and passes HTML through. `text` is the one that changes something: it escapes the characters that turn prose into markup, so a product description with a `50% *off*` in it arrives as its author wrote it rather than half-italic. ### Still data, too A mapped source keeps its `.ExternalData.` namespace. The records are pages **and** data, so a template that already iterated them still can. ## SQL (`type: sql`) Drivers: `mysql`, `mariadb`, `postgres`, `sqlite` (pure Go, no cgo). Queries live **only** in configuration — never in templates — and are statically validated: a single statement, `SELECT` (or `WITH … SELECT`) only, no piggybacked statements. Each query runs under the source `timeout` with a hard `max_rows` cap (default 10000); exceeding it is an error, not a silent truncation. DSNs are scrubbed from driver errors. ```yaml inventory: type: sql driver: mysql dsn: "$INVENTORY_DSN" queries: products: sql: | SELECT id, name, slug, price FROM products WHERE published = true max_rows: 10000 ``` ```gotemplate {{ range .ExternalData.inventory.products }}{{ .name }}: {{ .price }}{{ end }} ``` Operational rules: use a dedicated read-only database user, and TLS for remote connections (configure both in the DSN). ## CMS adapters (`type: cms`) ```yaml wordpress: type: cms adapter: wordpress # wordpress | drupal | movable_type driver: mysql dsn: "$WORDPRESS_DSN" mode: content # content (default) | data ``` `mode: content` merges the import into the site **before** URL, translation, taxonomy and collision processing — imported posts render, paginate, join feeds, sitemaps and archives exactly like native content. Imported authors merge without overwriting IDs the local metadata already defines. `mode: data` only exposes the import under `.ExternalData.` (pages/posts/authors/ taxonomies/media/metadata maps) — the data view is available in both modes. ### WordPress `wp_posts`, `wp_users`, `wp_terms`/`wp_term_taxonomy`/`wp_term_relationships`, `wp_postmeta` and attachments. `post` and custom post types render as posts; `page` maps to pages. `category`/`post_tag` feed the legacy fields; other taxonomies land in the page's taxonomies map, so [dynamic taxonomies](TAXONOMIES.md) pick them up. User-facing custom fields (keys not starting with `_`) land in `.Extra`. ```yaml wordpress: table_prefix: wp_ post_types: [post, page, guide] statuses: [publish] include_media: true include_custom_fields: true include_taxonomies: true ``` ### Drupal (8–11) `node_field_data`, `node__body`, `users_field_data`, `taxonomy_term_field_data`/`taxonomy_index` (vocabularies → taxonomies map) and `path_alias` — aliases are preserved as explicit links, so Drupal URLs survive the migration. With `include_fields: true`, dynamic `node__field_*` tables are discovered per engine and land in `.Extra`. Drupal 7 uses a different schema and is deferred. ```yaml drupal: version: 10 bundles: [article, page] published_only: true include_fields: true ``` ### Movable Type `mt_entry` (released entries and pages), `mt_author`, `mt_category`/`mt_placement`, `mt_tag`/`mt_objecttag`, `mt_asset` and — opt-in — visible `mt_comment` rows (GO-058). ```yaml movable_type: include_entries: true include_pages: true include_assets: true include_comments: true # visible comments → the page's .Extra["comments"] ``` With `include_comments: true` each imported entry carries its approved (`comment_visible = 1`) comments as a list of maps (author, email, url, created_on, text) under `.Extra["comments"]`, ready for templates. Unapproved and spam comments are never imported. Default `false` — identical behaviour to previous releases. ## Template API ```gotemplate {{ .ExternalData.products }} — parsed data per source {{ .ExternalDataMeta.products.FetchedAt }} — metadata (index and page contexts) {{ getExternal "products" }} — helper, works in every context {{ getExternalMeta "products" }} — Metadata struct ``` Metadata fields: `SourceType`, `Identifier` (always credential-free), `FetchedAt`, `FromCache`, `Stale`, `Checksum`, `RecordCount`, `ContentType`. ## Transformations ```yaml transform: select: data.items # dot path into the parsed structure ``` `select` unwraps API envelopes before the data reaches templates. There is deliberately no scripting, `eval` or embedded query runtime. ## Migration notes - `.Data` (local `data/` files) is unchanged; the new system is additive. - MDDB remains a separate content source; it may implement the same connector interface later. - Builds stay deterministic offline: `--offline` + a warmed cache produces byte-identical output. ## Troubleshooting - `failed at fetch … blocked address` — the host resolves to a private IP; set `allow_private: true` for self-hosted APIs. - `offline mode and no cached copy` — run once online, or set `fail_on_cache_miss: false` to downgrade the miss to a warning. - `must reference an environment variable` — secrets belong in the environment, not the config file. - `result exceeds max_rows` — raise the query's `max_rows` or narrow the SQL. ## Deferred (phase 7+) - Direct-URL template helpers (`getJSON`/`getCSV`/`getXML`). - Adapters: Ghost, Strapi, Contentful, Sanity, Notion, Airtable, Google Sheets, GitHub, GitLab; Drupal 7. - `watch: true` rebuilds on file-source changes. - Example CMS projects with seed scripts.