Overview

Last week I walked through the Argo CD and Kargo objects a golden-path layer generates, and closed on the thing that wasn’t there: not a single secret anywhere in those manifests. That wasn’t tidy example hygiene. It’s a design position, and this post defends it.

The tension is familiar. The GitOps pitch is everything in Git — desired state, reviewable, revertible, reconciled. The security pitch is never commit a secret. The most common resolution is to split the difference: encrypt the secret and commit the ciphertext, with Sealed Secrets or SOPS. It works, lots of good teams do it, and I think it’s the wrong default.

The better resolution is indirection: Git holds references, an external store holds values, and External Secrets Operator joins them inside the cluster. The interesting part is what the reference layer has to look like once real teams use it: values that differ per environment, per cluster, per app, owned by different people, merging in a predictable order. That’s where most secret setups quietly rot.

Why encrypted-in-Git is the wrong default

Three reasons, in increasing order of how much they’ve personally bitten me.

Git never forgets. Ciphertext in Git is a bet that the decryption key stays private forever — because the history does. Leak the key once and every secret ever committed is retroactively open, including the ones you “deleted” three years ago. A leaked store token, by contrast, reads the secrets that exist now, and revoking it ends the incident. With Git, the incident has been sitting in every clone all along.

Rotation fights the medium. Rotating an encrypted-in-Git secret means a commit; rotating the encryption key is worse. Sealed Secrets renews its sealing key every 30 days, but — as its own docs are careful to say — renewal does not re-encrypt existing sealed secrets; old ciphertext stays decryptable by old keys until you re-seal everything. So key hygiene becomes a repo-wide chore, and chores lose. The industry numbers say exactly this: GitGuardian’s State of Secrets Sprawl 2026 found 29 million new hardcoded secrets on public GitHub in 2025 alone — up 34% year over year — and that around 70% of credentials found valid in 2022 were still valid in January 2025. Nobody rotates what’s annoying to rotate.

A blob can’t tell you who read it. SOC 2 auditors ask questions Git has no answer for: who accessed the production database credential, when, and under what role? An external store answers from its audit log. A ciphertext file in a repo that fifty people can clone answers with a shrug.

One honest carve-out before moving on: encrypted-in-Git is fine — genuinely the right tool — for the handful of bootstrap secrets that exist before your secrets machinery does. The layer I’m describing uses a SealedSecret for exactly one thing per cluster: the token that lets the cluster talk to the external store. One sealed secret whose job is to get every other secret out of Git.

The shape: references in Git, values in a store

External Secrets Operator has been a CNCF project since 2022. Its job is exactly the indirection above: you commit an ExternalSecret — a description of which keys to fetch from which store — and the operator materializes and refreshes a plain Kubernetes Secret in-cluster from HashiCorp Vault, 1Password, a cloud secret manager, wherever.

So the golden-path layer’s job splits cleanly in two:

  • The write path never touches Git. When someone sets a secret — CLI or UI — the value goes over the layer’s API straight into the external store, versioned with check-and-set so concurrent writers can’t silently clobber each other. Nothing lands in the GitOps repo, nothing in an annotation, nothing in a CI log.
  • The read path is pure GitOps. What lands in the repo is an ExternalSecret per app per environment — names and paths only — which Argo CD syncs like any other manifest.

There’s a discipline hiding in that split that I’d call the load-bearing design rule: the control plane is value-blind. The layer’s storage interface can create items, upsert keys, list key names, and delete — it structurally cannot read values back out. Exactly one value-returning read exists in the entire secrets package, for offline backend migration, and nothing behind an HTTP handler ever holds a reference to it. You don’t audit your way out of exfiltration bugs; you make the type system refuse to compile them.

Scopes: two axes, and a merge you don’t perform

Here’s where real-world mess arrives. DATABASE_URL differs between staging and prod. SENTRY_DSN is one value for the whole org. A payments app has keys nobody else may share. During an incident, you need to override one value on one cluster without touching anything else.

Early designs (ours included) answer with a deep hierarchy — org, environment-type, project, app, app-in-environment, more. Every level looks reasonable in a design doc; in practice, deep hierarchies die of “where is this value actually coming from?” We collapsed ours to two axes:

  • Scope — where the value applies: global, per-env, or per-cluster, with project- and preview-level refinements inside those bands.
  • Tier — who owns it: shared (platform/org defaults) or app (one app’s own values).

Precedence is last-wins along one fixed order: global before env, env before cluster; within each band, shared before app. Cluster deliberately wins over everything — it’s the platform engineer’s escape hatch, the place a break-glass override or a per-cluster kill switch lives during an incident.

And now the part I find genuinely elegant: the layer doesn’t perform this merge. ESO does. An ExternalSecret applies its dataFrom entries in order, later keys overwriting earlier ones, into a single Kubernetes Secret. So the whole precedence model compiles down to the order of a YAML list. For app hello in project acme, prod environment, cluster prod-1 (labels anonymized as last week — platform.example/…, managed-by: your-platform):

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: hello-secrets
  namespace: hello-prod
  labels:
    app.kubernetes.io/managed-by: your-platform
    platform.example/app: hello
    platform.example/project: acme
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: platform-store
    kind: ClusterSecretStore
  target:
    name: hello-secrets
    creationPolicy: Owner
  dataFrom:
    - extract: { key: platform-secrets-global/shared-global }
    - extract: { key: platform-secrets-global/shared-project-acme }
    - extract: { key: platform-secrets-global/acme-hello-global }
    - extract: { key: platform-secrets-env-prod/shared-env-prod }
    - extract: { key: platform-secrets-env-prod/acme-hello-env-prod }
    - extract: { key: platform-secrets-env-prod/acme-hello-cluster-prod-1 }

Read it top to bottom and you’re reading the precedence rules: org defaults, then project defaults, then the app’s own globals, then the env band, and the cluster override last, winning. Debugging “where did this value come from” is reading a list. (Two mechanical notes: entries are only emitted for scope items that actually hold keys, because ESO errors on a missing item; and on Vault the item keys must be path-qualified exactly as above — get that wrong and ESO looks up a bare name at the mount root, finds nothing, and the ExternalSecret sits NotReady, silently.)

Application charts, meanwhile, never define these objects — they envFrom the expected Secret name with optional: true, so pods can start before ESO has filled it in. No sync-wave choreography.

Least privilege without a store per team

The obvious way to isolate teams is topology: a SecretStore per team or per environment, each with its own credentials. It works, and it multiplies the objects you generate, rotate, and debug — the small-team tax this series is about avoiding.

The layer does it with policy instead. Every cluster gets one ClusterSecretStore with one fixed name — which is what lets an app’s ExternalSecret stay byte-identical across clusters — but the token behind that store composes read policies per scope prefix, never mount-wide:

# one policy per scope prefix — this cluster is bound to prod, so it gets
# eso-read-global plus eso-read-env-prod, and nothing else
path "platform/data/platform-secrets-env-prod/*" {
  capabilities = ["read"]
}

A cluster bound only to staging physically cannot read prod values — which is the prod/non-prod segregation line a SOC 2 audit actually probes, enforced by the store rather than by promises. The only mount-wide write capability belongs to the control plane, and store tokens default to a 90-day TTL, on the theory that the exposure window should match the rotation cadence an operator will actually keep, not the longest one the store accepts.

One more behavior worth stating because its absence burned us: writes fail closed. If the configured backend is unreachable, reads degrade to a fallback store so running apps keep working, but writes are refused outright. An earlier build silently “helped” by writing to that fallback instead — the API returned 200, read-back looked correct, and the real vault was empty. That bug is why I now believe a secrets write path should be the least forgiving code you own.

What we don’t get for free

The honest section, as usual.

The tool this post stands on nearly stalled last year. In August 2025, ESO’s maintainers paused all releases — features, patches, images — citing burnout and a maintainer team far too small for the project’s adoption. It was widely covered as a case study in critical-path OSS running on fumes. The story since is genuinely good — new maintainers stepped up, and releases have been shipping on a regular cadence through 2026 — but the episode stays on the risk register: betting on ESO is betting on community sustainability, not just code quality. If ESO is in your production path, contribute — engineering time is worth more to that project than stars.

Policy granularity depends on the backend. The per-scope-prefix read policies above are a Vault feature. 1Password Connect tokens, for instance, are vault-scoped, not item-scoped — so two clusters bound to the same environment can each read the other’s cluster-level overrides. Acceptable for most teams, but it’s a real asymmetry, and pretending backends are interchangeable would be selling.

The layer computes policies; an operator applies them. Deliberately, the control plane’s token carries no policy-admin rights — it can’t escalate its own access. The cost is a manual step, and manual steps get skipped in a hurry. We still think it’s the right trade for a security boundary.

Indirection has a freshness lag. A rotated value propagates on the refresh interval (a minute, by default), and pods only see it per their reload behavior. Miles better than rotate-via-commit — but “instant” it is not.

Where this is going

Notice what the scope model quietly assumed: environments worth having scopes for — including ephemeral ones. Preview environments per change, release trains where stable and canary share a namespace, weighted traffic splitting on Gateway API HTTPRoutes, and promotion gates between all of it. That’s the next post: progressive delivery for teams without a progressive-delivery team.

Until then, a question for anyone running secrets-in-Git today: what does your rotation story actually look like in practice — not the wiki version, the real one? And if you’ve moved from encrypted-in-Git to reference-based, what broke on the way? The migration scars are the part nobody writes up, and the part I’d most like to compare notes on. If this series is useful, follow along — the next few posts are where it starts clicking together.