Overview

Last week’s post closed with a promise that was really four promises: preview environments per change, release trains where stable and canary share a namespace, weighted traffic splitting on Gateway API HTTPRoutes, and promotion gates between all of it. Honesty being the house rule of this series: two of those four are running in anger, and two are still designs. This post is about the real two — ephemeral previews and promotion — and ends with a straight account of the other two, because I’d rather correct myself than let a closing paragraph quietly become marketing.

Here’s the claim that survived contact with implementation. Preview environments are table stakes now; every PaaS spins up a URL per pull request. But most preview setups are a parallel universe: a different set of manifests, dummy secrets, a wildcard cert and a prayer. Whatever you validate there, you validated in the parallel universe — and then prod deploys through an entirely different pipeline. The property worth building is narrower and harder: a preview should be a clone of a real environment on the same rails — same chart, same GitOps repo, same secrets machinery — and the exact image you validated should be the thing that promotes.

A preview is a clone, not a snowflake

In the golden-path layer this series describes, an environment is a small, boring object: a name, an order in the promotion chain, a cluster binding. A preview is the same object with two differences: its order is zero — it sits outside the promotion chain, nothing auto-flows into or out of it — and it records a base environment, the stable env it clones. pr-42 isn’t a freestanding thing; it’s staging, forked.

Cloning buys determinism everywhere the parallel-universe approach improvises. The namespace comes from a pattern, {project}-{app}-preview-{name} by default, so app hello in project acme lands in acme-hello-preview-pr-42. The hostname is derived, not invented: pr-42.hello.preview.acme.com, built from the base env’s configured domain — the same one the chart renders — with a URL generated only when the app actually exposes an HTTP route — a queue worker’s preview gets pods, not a vanity URL. And the chart is the same chart staging runs, fed overlaid values rather than a values-preview.yaml that drifted six months ago.

The part I like most is what “creating a preview” mechanically is: committing files. The control plane writes a values.yaml and a small app.yaml descriptor under previews/staging/acme/pr-42/hello/ in the GitOps repo, plus the app’s ConfigMap and ExternalSecret descriptors in a parallel platform-resources tree — four files, and that’s the whole write. No Argo CD API call creates the Application; an ApplicationSet with a Git file generator watches that tree and manufactures one Application per descriptor:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: previews
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://git.example/acme/gitops.git
        revision: main
        files:
          - path: "previews/*/*/*/*/app.yaml"
  template:
    metadata:
      name: "{{appName}}-{{previewName}}"
      labels:
        app.kubernetes.io/managed-by: your-platform
        platform.example/env-type: preview
    spec:
      destination:
        server: "{{clusterServer}}"
        namespace: "{{namespace}}"
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [CreateNamespace=true]

Deleting a preview is pruning those files; the generator notices, the Application goes, the namespace goes. The whole lifecycle reduces to Git operations that Argo CD reconciles — which means the preview machinery inherits every property the rest of the platform already has: auditable history, revertability, and nothing to clean up by hand when a controller hiccups.

Config and secrets: overlays, not copies

Configuration for a preview resolves through three bands, later winning: the base env’s values, then a preview band — settings that apply to every preview of the app, the natural home for “one replica, no PDB, tiny resources” — then per-change overrides for the odd PR that needs a feature flag.

Secrets follow the same shape, and this is where last week’s post pays off. I wrote then that the scope model “quietly assumed environments worth having scopes for — including ephemeral ones.” Concretely: no per-preview vault is ever created. Preview secrets are just additional items inside the base env’s existing store — a shared hello-env-preview band item, plus hello-env-preview-pr-42 when one PR needs its own value — read through the same ClusterSecretStore and the same per-scope-prefix policies staging already uses. Twenty open PRs cost zero new stores, zero new tokens, zero new policies to rotate. The ExternalSecret in the preview namespace is the same list-of-extracts you saw last week, two entries longer.

The trigger is your CI, on purpose

There’s no PR-webhook receiver in the platform. The API is two endpoints, and your CI drives them:

on:
  pull_request:
    types: [opened, synchronize, reopened, closed]

jobs:
  preview:
    steps:
      # build & push image tagged with the PR head SHA, then:
      - name: Create or update preview
        if: github.event.action != 'closed'
        run: |
          curl -fsS -X POST "$PLATFORM_API/projects/acme/apps/hello/previews" \
            -H "Authorization: Bearer $TOKEN" \
            -d "{\"name\":\"pr-$PR\",\"imageTag\":\"$SHA\"}"
      - name: Delete preview
        if: github.event.action == 'closed'
        run: |
          curl -fsS -X DELETE "$PLATFORM_API/projects/acme/apps/hello/previews/pr-$PR"

The create is an upsert: the first call returns 201, every subsequent push re-POSTs with the new immutable tag and gets a 200 — the preview re-points, Argo CD rolls it. Two design details here earn their keep. Deletion prunes the Git files first and only then drops the record, deliberately ordered so a Git failure leaves the preview listed and retryable instead of orphaning running workloads behind a deleted record. And keeping SCM integration out of the control plane is a real trade, not laziness: the platform stays webhook-agnostic across GitHub, GitLab, and whatever’s next, at the cost that the preview lifecycle is exactly as reliable as the CI job that drives it. Forget the closed handler and nothing reaps the leak for you.

Promotion is a commit

Now the other half of the title. Deploying to prod, in this model, is not a pipeline run with a colored button — it’s a Git commit that a promotion engine writes for you. The layer generates Kargo objects per app: a Warehouse that watches the image registry, and a Stage per stable environment, chained by order.

apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: hello
  namespace: kargo-acme
spec:
  subscriptions:
    - image:
        repoURL: registry.example/acme/hello
        allowTags: "^[0-9a-f]{7}$"        # 7-char git SHAs only
        imageSelectionStrategy: NewestBuild

Each Stage carries a promotion template, and reading it tells you exactly what a promotion does — no more, no less:

promotionTemplate:
  spec:
    steps:
      - uses: git-clone
        config: { repoURL: https://git.example/acme/gitops.git,
                  checkout: [{ branch: main, path: ./src }] }
      - uses: yaml-update
        config:
          path: ./src/envs/prod/acme/hello/values.yaml
          updates:
            - key: image.tag
              value: ${{ quote(imageFrom("registry.example/acme/hello").Tag) }}
      - uses: git-commit
        config: { path: ./src, message: "promote acme/hello to prod" }
      - uses: git-push
        config: { path: ./src }
      - uses: argocd-update
        config: { apps: [{ name: acme-hello-prod-1 }] }

Clone, edit one YAML key, commit, push, wait for Argo CD health. The promotion is the commit — reviewable in history, revertible with the tools you already have. (One scar embedded in that template: the quote() matters. Seven hex characters means a tag can be all digits — 4211884 round-trips through YAML as an integer, 123e456 as a float in scientific notation — and Helm then rejects the non-string image.tag with a type error. This class of bug is exactly why “a promotion is a YAML edit” needs to be engineered, not scripted; the generated template requires Kargo ≥ v1.3.4, where quoting behaves correctly.)

The gates are policy, not folklore: the first stage auto-promotes whatever the Warehouse admits; downstream stages require a manual promote call unless the app opts into auto-promotion; and any stage can be pinned — frozen at its current version, auto-promotion suspended, which is the two-line answer to “it’s Diwali weekend, nothing moves.”

And previews close the loop from the top of this post: a preview can promote to any stable environment. Mechanically it’s a pin — prod frozen at the preview’s image tag — which means the exact SHA-tagged image your reviewers clicked around on Tuesday is the image prod runs on Thursday, not a rebuild that’s probably the same. Rollback is the same machinery pointed backwards: re-promote a version from the stage’s own recorded history, with the engine refusing any version that stage never actually ran.

What we don’t get for free

The honest section, and this week it has teeth.

Release trains and weighted canaries are not built. That’s half of last week’s promise, and I’m retracting it until it ships. The design exists — stable and canary trains in one namespace, traffic split by weighted backendRefs on a Gateway API HTTPRoute — but no code implements it: no train model, no weight anywhere, and a UI tab that says “coming soon” in so many words. What exists today is honest plumbing: the platform exposes the cluster’s Gateway as tokens your own chart can reference, and reads HTTPRoutes back to build app URLs. If you need a weighted canary right now, you author the route rules in your chart and manage weights by hand; the platform won’t fight you, and won’t help you either. I’d rather tell you that than demo YAML that only exists in a roadmap file.

The promotion gate is health, not analysis. There are no metric-driven verification steps — no “hold the promotion until error rate stays under 1% for ten minutes.” The de-facto gate is Argo CD reporting the upstream environment healthy. A consequence worth stating plainly: because stages define no verification, Kargo’s own freight-availability flow doesn’t engage, and the layer runs its own small reconciler — walking envs in order, requiring upstream health, one step per tick with a failure cooldown — and explicitly approves freight for the target stage. It works, and it’s duplicated machinery standing in for verification gates that should eventually exist.

Preview lifecycles trust your CI. No webhook receiver also means no safety net below CI. The closed handler is the garbage collector.

Multi-component apps take a simpler path. An app composed of several components previews with pinned tags and skips the Warehouse/Stage machinery entirely — a real asymmetry between the simple case and the composed one.

Where this is going

Everything in this post — the ApplicationSet, the preview descriptors, the Warehouses and Stages, the promotion commits — lives as plain Argo CD and Kargo objects in a Git repo you own, labeled but not renamed, readable without any vendor tooling. Which sets up the next post’s uncomfortable question: if the platform disappeared tomorrow, would your apps keep running — and could you keep operating them with nothing but git and kubectl? Lock-in isn’t a pricing-page problem, it’s an architecture problem, and I want to show what designing for leaving looks like.

Until then: what does “promotion” actually mean where you work — a Git commit, a Jenkins job, a person with production kubectl and good intentions? And do your preview environments share rails with prod, or live in the parallel universe? I keep collecting these stories, and the gap between the wiki version and the real one is where all the interesting engineering hides. If the series is useful, follow along — next week is the one I’ve been wanting to write since post one.