Before a cluster runs its first app, a small team usually adds the same
things: certificates, an admission policy, metrics and dashboards, a way to
get secrets in, backups, and a tool that deploys from Git. Each of them is
an operator that reads custom resources. Those resources name each other.
A Certificate names a ClusterIssuer, an ExternalSecret names a
ClusterSecretStore, and a Kyverno policy selects namespaces by their
labels. Kept in separate charts and manifests, each of these names is
typed again in every file that uses it.
This post keeps those resources in one Kapitan
inventory, where each shared name is set once. The
project is
platform-baseline
in kadet-hub/examples. Its check script deploys it to a new
kind cluster on every change and once a week, and fails
when one of the steps below stops working. podinfo, a small demo web
server, runs on top and uses every part, so the checks go through an app
and not through the operators' own status.
What the inventory renders
kadet-hub's generators render the custom resources
an operator reads. They do not install the
operator. kadet-hub publishes a recipe for
cert-manager, which renders the controller, webhook and cainjector with
the kubernetes generator. For the other
operators there is no recipe, so check applies the release manifest of
each project, pinned by version and sha256.
| Component | Operator, installed from | The inventory renders |
|---|---|---|
| cert-manager 1.21.2 | recipe cert-manager 1.0.3 | a self-signed root CA and a CA ClusterIssuer, with certmanager 1.0.4 |
| Kyverno 1.19.2 | install.yaml | one ValidatingPolicy, with kyverno 1.1.4 |
| Prometheus Operator 0.94.2 | bundle.yaml | a Prometheus and its RBAC, with prometheus-operator 1.1.6 |
| Grafana Operator 5.25.0 | kustomize-cluster_scoped.yaml | a Grafana and its Prometheus data source, with grafana 1.0.3 |
| External Secrets 2.14.0 | external-secrets.yaml | a ClusterSecretStore, with external-secrets 1.0.6, and the namespace and RBAC it reads |
| Velero 1.18.4 | velero install | a BackupStorageLocation and a Schedule, with velero 1.0.2 |
| Argo CD 3.5.6 | install.yaml | an AppProject and one Application per app, with argocd 1.0.4 |
The release manifests of the Prometheus Operator and of External Secrets
install into the namespace default. Moving them takes an edited copy of
the manifest or a Helm chart rendered for another namespace, for example
through Kapitan's helm input. This project does neither and leaves them
in default.
Kapitan in brief
A Kapitan inventory holds classes and
targets, both YAML. A target lists the classes it
includes, and Kapitan merges their parameters in that order, the target's
own last. ${platform.domain} is an
interpolation: it takes the value of that key
from the merged parameters of the target. kapitan compile writes each
target into compiled/<target>/. A generator is a Python library that
reads one section of the parameters, such as generators.certmanager, and
writes manifests. The
first platform guide shows the setup from an empty
directory.
This project has one target per component, cert-manager, kyverno,
monitoring, external-secrets, velero and argocd, and the target
podinfo for the app. All of them include two classes. generators pins
the generators by version and digest. platform
holds the values that resources in more than one target read. The recipe
is written for the omegaconf inventory backend, which .kapitan selects:
global: inventory-backend: omegaconf Fetch and verify
Kapitan 0.36.3 reads the inventory before it fetches dependencies, so the
cert-manager recipe is pinned in a separate fetch inventory,
fetch/targets/recipes.yml:
parameters: kapitan: dependencies: - type: oci source: ghcr.io/kadet-hub/recipes/cert-manager:1.0.3@sha256:99d27b9b0ed0afa330d48e1b12cc0bee25f37c5a4b5129ba07a68dfe5063bbef output_path: inventory/classes/recipes/cert-manager check does not compile it with --fetch, which checks neither digests
nor attestations. It runs the steps of the
registry's
consumer docs
instead. For the recipe:
id=https://github.com/kadet-hub/registry/.github/workflows/release.yml@refs/heads/main gh release download --repo kadet-hub/registry --pattern 'index.json*' gh attestation verify index.json --bundle index.json.sigstore.json --owner kadet-hub --cert-identity "$id" d=sha256:99d27b9b0ed0afa330d48e1b12cc0bee25f37c5a4b5129ba07a68dfe5063bbef jq --arg d "$d" '.recipes["cert-manager"] | to_entries[] | select(.value.digest == $d) | {version: .key, yanked: .value.yanked}' index.json gh attestation verify "oci://ghcr.io/kadet-hub/recipes/cert-manager@$d" --owner kadet-hub --cert-identity "$id" tmp=$(mktemp -d) oras pull -o "$tmp" "ghcr.io/kadet-hub/recipes/cert-manager@$d" mkdir -p inventory/classes/recipes/cert-manager tar -xf "$tmp/cert-manager.tar" -C inventory/classes/recipes/cert-manager --no-same-owner The first attestation check proves that the registry's release workflow
signed the index. The jq line must print version
1.0.3 and yanked: null. The second attestation check proves the
release workflow published the recipe at that digest. The generators go
through the same steps against .generators in the index and unpack into
lib/<name>. inventory/classes/generators.yml pins seven of them:
parameters: kapitan: dependencies: - type: oci source: ghcr.io/kadet-hub/certmanager:1.0.4@sha256:99bfa0edf76fd7f564c101ce7dedbd2e0a790825db0173f1a64faed51bc0fd1f output_path: lib/certmanager - type: oci source: ghcr.io/kadet-hub/kyverno:1.1.4@sha256:af3f1376bdc69e261bb75d23c01014a24926e14f52780c86d8e941c524525de6 output_path: lib/kyverno - type: oci source: ghcr.io/kadet-hub/prometheus-operator:1.1.6@sha256:99faa008d1335edc787f164685a04e855012b6b48364e67737fb764bb98c449b output_path: lib/prometheus-operator - type: oci source: ghcr.io/kadet-hub/external-secrets:1.0.6@sha256:184e8a60f92c0c09eae32973ceaf909b2da53457e88233161ffb90c4b9a63e60 output_path: lib/external-secrets - type: oci source: ghcr.io/kadet-hub/velero:1.0.2@sha256:95c56a6cf87a55863b987e58fd2d21cea24c67b91ff9bac2fec2d796c8c8f46b output_path: lib/velero - type: oci source: ghcr.io/kadet-hub/argocd:1.0.4@sha256:8b41817b189cdc043bb1a726f0e42a827672a1eda80e8d8406b512262182feea output_path: lib/argocd - type: oci source: ghcr.io/kadet-hub/grafana:1.0.3@sha256:a31a55e6eb06eeb6c28ed61e12b8971b0c23ab446eb3f8d7e43bd1b290cbf836 output_path: lib/grafana The eighth, kubernetes 1.14.0, comes from the recipe's own classes. A
second pin of it would unpack into the same lib/kubernetes, so check
fails when one generator is pinned at two digests. The targets
monitoring, external-secrets and podinfo use the kubernetes
generator too, and get the version the recipe pins.
The shared class
inventory/classes/platform.yml:
parameters: platform: # App hostnames are <app>.<domain>. domain: apps.example.com # The ClusterIssuer that signs the apps' certificates, and its CA. issuer: platform-ca # The ClusterSecretStore that apps read their secrets from. secret_store: platform namespaces: cert_manager: cert-manager monitoring: monitoring # Holds the secrets the store serves; Velero backs it up. secrets: platform-secrets # The repository Argo CD deploys the apps from. git_repo: http://gitea.git.svc.cluster.local:3000/kadet/apps.git # Argo CD labels every app namespace it creates with these, and the # Kyverno policy applies to the namespaces that carry them. tenant_labels: platform.example.com/tenant: "true" pod-security.kubernetes.io/enforce: restricted | Value | Read by |
|---|---|
domain | the root CA's common name in cert-manager, podinfo's DNS name in podinfo |
issuer | the CA ClusterIssuer and its CA Certificate in cert-manager, the issuerRef of podinfo's Certificate |
secret_store | the ClusterSecretStore in external-secrets, the store of podinfo's ExternalSecret |
namespaces.cert_manager | the recipe's namespace and the CA Certificate in cert-manager |
namespaces.monitoring | Prometheus, Grafana and the data source URL in monitoring |
namespaces.secrets | the store's namespace, ServiceAccount and Role in external-secrets, the Schedule in velero |
git_repo | the AppProject and the Applications in argocd |
tenant_labels | the namespace labels in argocd, the namespace selector in kyverno |
The label pod-security.kubernetes.io/enforce: restricted makes the API
server enforce the restricted Pod Security profile in every app namespace.
The other label exists for the Kyverno selector. Both live in one map, so
the selector matches exactly the namespaces Argo CD creates.
The platform targets
cert-manager
inventory/targets/cert-manager.yml:
classes: - generators - platform - recipes.cert-manager - recipes.cert-manager.crds parameters: namespace: ${platform.namespaces.cert_manager} kapitan: vars: target: cert-manager compile: - {input_type: kadet, output_type: yml, input_paths: [lib/certmanager], output_path: issuers} generators: kubernetes: namespaces: cert-manager: name: ${platform.namespaces.cert_manager} certmanager: # A self-signed root CA and the ClusterIssuer that signs with it. ca_issuers: platform: name: ${platform.issuer} namespace: ${platform.namespaces.cert_manager} common_name: ${platform.domain} root CA The two recipe classes add the operator and its CRDs, compiled into
compiled/cert-manager/manifests. The recipe renders no Namespace, so the
target adds one through the kubernetes generator. The compile entry adds the
certmanager generator, whose output goes to
compiled/cert-manager/issuers. check applies the second directory
only once cert-manager's webhook runs. One ca_issuers entry renders three
objects. The ClusterIssuer platform-ca-selfsigned signs the CA
Certificate platform-ca, an ECDSA P-256 key and certificate that
cert-manager stores in the Secret platform-ca. The ClusterIssuer
platform-ca signs app certificates with that CA. A ClusterIssuer reads
its CA Secret from cert-manager's own namespace, which is why the entry
sets namespace.
Kyverno
inventory/targets/kyverno.yml:
classes: - generators - platform parameters: kapitan: vars: target: kyverno compile: - {input_type: kadet, output_type: yml, input_paths: [lib/kyverno], output_path: .} generators: kyverno: validating_policies: require-image-digest: match_kinds: [Pod] match_constraints: namespaceSelector: matchLabels: ${platform.tenant_labels} validation_actions: [Deny] validations: - expression: >- object.spec.containers.all(c, c.image.contains('@sha256:')) && (!has(object.spec.initContainers) || object.spec.initContainers.all(c, c.image.contains('@sha256:'))) message: Images in app namespaces must be pinned by digest. A ValidatingPolicy is Kyverno's policy kind with rules in CEL, the
expression language of Kubernetes' own admission policies. Deny rejects a
violating request, which is Kyverno's enforce mode. The policy requires
every container image in an app namespace to carry a digest, because a tag
can be moved to other content and a digest cannot. match_kinds: [Pod] is
a generator shorthand for the rule on pods with the operations CREATE
and UPDATE. Kyverno derives the same rule for Deployments, StatefulSets,
DaemonSets, ReplicaSets, Jobs and CronJobs, as the policy's status lists.
check tests the rule on pods only. Namespaces without the tenant labels
are not checked, so the operators' own images, which their manifests name
by tag, still run.
Prometheus and Grafana
inventory/targets/monitoring.yml:
classes: - generators - platform parameters: namespace: ${platform.namespaces.monitoring} kapitan: vars: target: monitoring compile: - {input_type: kadet, output_type: yml, input_paths: [lib/kubernetes], output_path: .} - {input_type: kadet, output_type: yml, input_paths: [lib/prometheus-operator], output_path: .} - {input_type: kadet, output_type: yml, input_paths: [lib/grafana], output_path: .} generators: kubernetes: namespaces: monitoring: name: ${platform.namespaces.monitoring} prometheus_operator: prometheuses: platform: rbac: true retention: 2d service_monitor_selector: {} service_monitor_namespace_selector: {} resources: requests: {cpu: 100m, memory: 256Mi} limits: {memory: 1Gi} grafana: instance_selector: grafana: platform instances: platform: labels: grafana: platform datasources: prometheus: uid: prometheus datasource: name: Prometheus type: prometheus access: proxy url: http://prometheus-operated.${platform.namespaces.monitoring}.svc:9090 isDefault: true rbac: true adds the ServiceAccount, ClusterRole and ClusterRoleBinding
that the Prometheus Operator's RBAC guide lists. retention: 2d keeps
two days of samples. The two empty selectors
pick every ServiceMonitor in every namespace. An unset selector would pick
none. The Grafana Operator imports a GrafanaDatasource into every Grafana
instance whose labels match instance_selector. prometheus-operated is
the headless Service the Prometheus Operator creates in each namespace
that has a Prometheus.
External Secrets with the Kubernetes provider
inventory/targets/external-secrets.yml:
classes: - generators - platform parameters: namespace: ${platform.namespaces.secrets} kapitan: vars: target: external-secrets compile: - {input_type: kadet, output_type: yml, input_paths: [lib/kubernetes], output_path: .} - {input_type: kadet, output_type: yml, input_paths: [lib/external-secrets], output_path: .} generators: kubernetes: namespaces: secrets: name: ${platform.namespaces.secrets} service_accounts: store-reader: {} rbac: store-reader: namespace: ${platform.namespaces.secrets} service_accounts: [store-reader] rules: - {apiGroups: [""], resources: [secrets], verbs: [get, list, watch]} external_secrets: clustersecretstores: platform: name: ${platform.secret_store} spec: provider: kubernetes: remoteNamespace: ${platform.namespaces.secrets} server: caProvider: type: ConfigMap name: kube-root-ca.crt key: ca.crt namespace: ${platform.namespaces.secrets} auth: serviceAccount: name: store-reader namespace: ${platform.namespaces.secrets} The store uses External Secrets' Kubernetes provider. Through it,
External Secrets reads Secrets from the namespace platform-secrets with
a token of the ServiceAccount store-reader, which may read Secrets there
and nowhere else.
kube-root-ca.crt is the ConfigMap with the API server's CA that
Kubernetes puts into every namespace. The provider was picked because CI
can run it without an outside service. For Vault or a cloud secret
manager, the ClusterSecretStore gets another provider block. The
ExternalSecret below keeps its store reference, and its remoteRef keys
follow the new provider's layout. The fake provider would also run in CI,
but it serves values written into the store itself, so no sync from
another place would be tested.
Velero with S3-compatible storage
inventory/targets/velero.yml:
classes: - generators - platform parameters: kapitan: vars: target: velero compile: - {input_type: kadet, output_type: yml, input_paths: [lib/velero], output_path: .} generators: velero: backup_storage_locations: default: provider: aws bucket: velero config: region: us-east-1 s3ForcePathStyle: "true" s3Url: http://s3.s3.svc:7070 credential: name: velero-s3 key: cloud schedules: secrets: schedule: CRON_TZ=UTC 0 3 * * * skip_immediately: true template: included_namespaces: ["${platform.namespaces.secrets}"] ttl: 168h0m0s Velero writes backups to S3-compatible storage through its AWS plugin.
s3ForcePathStyle puts the bucket into the URL path, which servers
without per-bucket host names need. The generator marks the only location
as the default. The Schedule backs up the secrets namespace daily at 03:00
UTC and keeps each backup for seven days (168h0m0s). skip_immediately
stops Velero from running a backup the moment the Schedule is created.
The Secret velero-s3 with the keys is not in the inventory. check
creates it at run time, as the keys of a real bucket would come from
outside the repository.
The S3 server is
Versity's S3 gateway 1.10.0
(Apache-2.0), in s3-server.yaml next to the inventory. Its posix backend
stores a bucket as a directory, so mkdir creates Velero's bucket. MinIO
was the first candidate. On 4 November 2026 neither quay.io/minio/minio
nor docker.io/minio/minio listed its tags to an anonymous client, so
there was no image to pin by digest.
Argo CD
inventory/targets/argocd.yml:
classes: - generators - platform parameters: kapitan: vars: target: argocd topics: argocd: consume: true compile: - {input_type: kadet, output_type: yml, input_paths: [lib/argocd], output_path: .} generators: argocd: projects: apps: source_repos: ["${platform.git_repo}"] destinations: - {namespace: "*", server: https://kubernetes.default.svc} # The namespace that CreateNamespace=true creates; no other # cluster-scoped kind. cluster_resource_whitelist: - {group: "", kind: Namespace} topics: argocd: project: apps source: repoURL: ${platform.git_repo} targetRevision: main directory: recurse: true sync_policy: automated: {prune: true, selfHeal: true} syncOptions: [CreateNamespace=true] managedNamespaceMetadata: labels: ${platform.tenant_labels} A Kapitan topic lets one target read parameters that other targets
publish. Every target that publishes argocd gets one Application, with
the values under topics.argocd as defaults. An app team then adds a
target, and the target argocd stays unchanged. Each Application syncs
automatically, prunes what Git no longer holds and undoes manual changes.
CreateNamespace=true creates the app's namespace with the tenant labels
from managedNamespaceMetadata. Argo CD refused that namespace until
Namespace was in the project's cluster_resource_whitelist. Without
directory.recurse it read only the top level of the app's path. The
generators write one directory per namespace below it, so Argo CD found no
manifest and still reported the Application Synced and Healthy.
The app
inventory/targets/podinfo.yml:
classes: - generators - platform parameters: namespace: podinfo kapitan: vars: target: podinfo topics: argocd: parameters: source: path: podinfo destination: ${namespace} compile: - {input_type: kadet, output_type: yml, input_paths: [lib/kubernetes], output_path: .} - {input_type: kadet, output_type: yml, input_paths: [lib/certmanager], output_path: .} - {input_type: kadet, output_type: yml, input_paths: [lib/external-secrets], output_path: .} - {input_type: kadet, output_type: yml, input_paths: [lib/prometheus-operator], output_path: .} generators: manifest: default_config: recommended_labels: true security_profile: restricted certmanager: certs: # No sync wave: the generator renders no annotations, and a missing # Secret volume only holds the pod in ContainerCreating. podinfo-tls: spec: secretName: podinfo-tls issuerRef: {name: "${platform.issuer}", kind: ClusterIssuer} dnsNames: ["podinfo.${platform.domain}"] external_secrets: externalsecrets: # Before the Deployment, whose env reads the Secret. podinfo-message: annotations: argocd.argoproj.io/sync-wave: "-1" cluster_store: ${platform.secret_store} refresh_interval: 10s spec: data: - secretKey: message remoteRef: {key: podinfo, property: message} prometheus_operator: service_monitors: podinfo: selector: {name: podinfo} endpoints: http components: podinfo: image: ghcr.io/stefanprodan/podinfo:6.15.0@sha256:ec73780a8425f59ea49f5bc8cdff0d598805a224fbaa1f86c67a244f250fa9da command: [./podinfo, --port=8080, --secure-port=8443, --cert-path=/data/cert] security_context: {runAsUser: 65534} ports: http: {container_port: 8080, service_port: 80} https: {container_port: 8443, service_port: 443} env: PODINFO_UI_MESSAGE: secretKeyRef: {name: podinfo-message<...
