If you have been running OpenShift Logging for a few years, you probably still have a three-node Elasticsearch cluster chewing through block storage, with a ClusterLogging CR that nobody dares to touch. Red Hat deprecated the Elasticsearch-backed stack in Logging 5.9 and removed it in Logging 6.0. The replacement is LokiStack: Grafana Loki deployed and managed by the Loki Operator, with chunks stored in object storage instead of PVCs.

This is not just a forced migration. It is the single easiest storage cost reduction available on an OpenShift cluster. This article walks through a real migration: bucket provisioning, LokiStack sizing, retention per tenant, ingestion limits, the ClusterLogForwarder with Vector, and the LogQL you will actually type at 3 a.m.

Why the Elasticsearch stack became the problem

Where the money goes

Elasticsearch stores an inverted index over every field of every log line, plus doc values, plus at least one replica shard. On OpenShift that index lives on PVCs — gp3, managed-csi, Ceph RBD — billed at roughly $0.08–0.12 per GB-month, provisioned 24/7 whether full or empty, and you over-provision by 25–30% because you cannot shrink a PVC.

Take a mid-size cluster ingesting 300 GB/day of raw logs with 7-day retention:

  • Elasticsearch: ~0.9x on-disk after compression, x2 for one replica, +25% headroom → roughly 4.7 TB of block storage, around $400/month on gp3, plus three JVM nodes with 16–32 GB heap each.
  • Loki: chunks are gzip/snappy-compressed blobs with a tiny index; expect 0.12–0.20x of raw volume → roughly 300–450 GB in S3, under $15/month, plus a few modest PVCs for the ingester WAL and index cache.

The ratio is not 2x, it is closer to 10x on storage and 2–3x on compute. The trade-off is real and you must accept it: Loki does not index log content. You filter by labels first, then grep. If your workflow is "full-text search across 30 days with aggregations", Loki will frustrate you. If it is "give me the logs of this namespace around 14:32", Loki is faster and far cheaper.

Target architecture

Vector (DaemonSet) -> lokistack-gateway (auth + tenant routing) -> Loki distributor | ingester -> S3/ODF bucket (chunks + index) querier <- compactor (retention) 

Three tenants are created automatically in openshift-logging mode: application, infrastructure, audit. Authentication and authorization go through the gateway, which validates the OpenShift bearer token via SubjectAccessReview against the loki.grafana.com API group. No shared password, no basic auth.

Step 1 — provision the object bucket

With ODF / NooBaa via ObjectBucketClaim

If you run OpenShift Data Foundation, an ObjectBucketClaim is the fastest path and keeps the data on-cluster.

apiVersion: objectbucket.io/v1alpha1 kind: ObjectBucketClaim metadata: name: loki-bucket namespace: openshift-logging spec: generateBucketName: loki-bucket storageClassName: openshift-storage.noobaa.io 

The OBC produces a ConfigMap and a Secret of the same name. Loki expects different key names, so convert them:

oc apply -f obc-loki.yaml BUCKET=$(oc get cm loki-bucket -n openshift-logging -o jsonpath='{.data.BUCKET_NAME}') HOST=$(oc get cm loki-bucket -n openshift-logging -o jsonpath='{.data.BUCKET_HOST}') PORT=$(oc get cm loki-bucket -n openshift-logging -o jsonpath='{.data.BUCKET_PORT}') AKID=$(oc get secret loki-bucket -n openshift-logging -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d) SKEY=$(oc get secret loki-bucket -n openshift-logging -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d) oc create secret generic logging-loki-s3 -n openshift-logging \ --from-literal=bucketnames="$BUCKET" \ --from-literal=endpoint="https://${HOST}:${PORT}" \ --from-literal=access_key_id="$AKID" \ --from-literal=access_key_secret="$SKEY" 

Because the NooBaa endpoint uses a service-CA certificate, reference the CA bundle in the LokiStack (spec.storage.tls.caName).

With AWS S3 and STS

On AWS, skip static keys entirely. Create the bucket, then a secret holding only the role ARN and let the Loki Operator mint short-lived tokens through the Cloud Credential Operator:

aws s3api create-bucket --bucket acme-openshift-loki --region eu-west-3 \ --create-bucket-configuration LocationConstraint=eu-west-3 aws s3api put-bucket-encryption --bucket acme-openshift-loki \ --server-side-encryption-configuration \ '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms"}}]}' aws s3api put-public-access-block --bucket acme-openshift-loki \ --public-access-block-configuration \ BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true oc create secret generic logging-loki-s3 -n openshift-logging \ --from-literal=bucketnames=acme-openshift-loki \ --from-literal=region=eu-west-3 \ --from-literal=role_arn=arn:aws:iam::123456789012:role/openshift-loki 

Then set spec.storage.secret.credentialMode: token in the LokiStack.

Add one lifecycle rule for safety — abort incomplete multipart uploads after 7 days. Do not add an expiration rule on objects: the Loki compactor owns retention, and an S3 lifecycle rule deleting chunks under Loki's feet produces unreadable streams.

Step 2 — install the Loki Operator and size the LokiStack

apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-operators-redhat namespace: openshift-operators-redhat spec: {} --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: loki-operator namespace: openshift-operators-redhat spec: channel: stable-6.2 name: loki-operator source: redhat-operators sourceNamespace: openshift-marketplace installPlanApproval: Automatic 

Choosing the t-shirt size

The operator exposes fixed sizes, each one a tested combination of replicas, CPU/memory requests and PVC sizes. Pick by daily ingested volume, not by node count:

Size Ingestion/day Typical use
1x.demo — lab only, no resource requests, not supported in prod
1x.pico up to 50 GB small cluster, single-tenant workloads
1x.extra-small up to 100 GB dev/test clusters
1x.small up to 500 GB most production clusters
1x.medium up to 2 TB large multi-tenant platforms

Measure before guessing. If Elasticsearch is still running, look at index growth per day. After Loki is live, the authoritative number is:

sum(rate(loki_distributor_bytes_received_total{namespace="openshift-logging"}[5m])) * 86400 / 1e9 

Sizes are upgrade-only in practice: going from 1x.small to 1x.medium is supported, shrinking is not. Start one notch below your estimate if you are unsure, then watch for 429s.

The LokiStack manifest

apiVersion: loki.grafana.com/v1 kind: LokiStack metadata: name: logging-loki namespace: openshift-logging spec: managementState: Managed size: 1x.small storage: schemas: - version: v13 effectiveDate: "2024-10-01" secret: name: logging-loki-s3 type: s3 tls: caName: openshift-service-ca.crt # ODF/NooBaa only  storageClassName: gp3-csi replicationFactor: 1 tenants: mode: openshift-logging limits: global: retention: days: 7 ingestion: ingestionRate: 10 ingestionBurstSize: 20 perStreamRateLimit: 5 perStreamRateLimitBurst: 15 maxGlobalStreamsPerTenant: 25000 maxLabelNamesPerSeries: 20 queries: queryTimeout: 3m maxChunksPerQuery: 2000000 maxEntriesLimitPerQuery: 5000 tenants: application: retention: days: 7 streams: - days: 30 priority: 1 selector: '{kubernetes_namespace_name="payments-prod"}' infrastructure: retention: days: 15 audit: retention: days: 30 

Two details that matter. storageClassName is for the ingester WAL, compactor and index cache PVCs only — it is small, but it must be a block storage class with decent IOPS. And retention is only enforced if spec.limits.global.retention exists; without it the compactor never deletes and your "cheap" bucket grows forever.

The streams block is the killer feature: per-namespace retention overrides, evaluated by priority. Keep 30 days for the regulated namespace and 7 days for everything else, in the same tenant.

Step 3 — ingestion limits and backpressure

ingestionRate is in MB/s per tenant, perStreamRateLimit in MB/s per stream (a stream = one unique label-set, in practice one container). The defaults are deliberately low. When a chatty pod blows past them, Vector gets HTTP 429 and you see this in the collector logs:

oc logs -n openshift-logging ds/collector -c collector | grep -i "429\|rate limit" 

Three valid reactions, in order of preference: fix the application's log level, drop the noise in the forwarder (next section), then raise the limit. Raising perStreamRateLimit to 10–15 MB/s is reasonable; raising maxGlobalStreamsPerTenant without thinking is how you OOM the ingesters, since every stream holds an in-memory chunk.

Step 4 — ClusterLogForwarder with Vector

In Logging 6.x the ClusterLogging CR is gone. Everything lives in a ClusterLogForwarder under observability.openshift.io/v1, and the collector runs under a service account you create and authorize explicitly.

oc create ns openshift-logging oc create sa collector -n openshift-logging oc adm policy add-cluster-role-to-user collect-application-logs -z collector -n openshift-logging oc adm policy add-cluster-role-to-user collect-infrastructure-logs -z collector -n openshift-logging oc adm policy add-cluster-role-to-user collect-audit-logs -z collector -n openshift-logging 
apiVersion: observability.openshift.io/v1 kind: ClusterLogForwarder metadata: name: collector namespace: openshift-logging spec: serviceAccount: name: collector managementState: Managed outputs: - name: default-lokistack type: lokiStack lokiStack: target: name: logging-loki namespace: openshift-logging authentication: token: from: serviceAccount tls: ca: key: service-ca.crt configMapName: openshift-service-ca.crt filters: - name: drop-probes type: drop drop: - test: - field: .message matches: '"GET /healthz|/readyz|/livez"' - name: prune-noisy-fields type: prune prune: notIn: - .message - .log_type - .kubernetes - .@timestamp - .level pipelines: - name: app-and-infra inputRefs: [application, infrastructure] filterRefs: [drop-probes, prune-noisy-fields] outputRefs: [default-lokistack] - name: audit inputRefs: [audit] outputRefs: [default-lokistack] 

Check that it rolled out and that logs are flowing:

oc get clusterlogforwarder collector -n openshift-logging -o jsonpath='{.status.conditions}' | jq oc get pods -n openshift-logging -l app.kubernetes.io/component=collector oc get lokistack logging-loki -n openshift-logging -o jsonpath='{.status.conditions}' | jq 

Finally, enable the console log viewer (Cluster Observability Operator must be installed):

apiVersion: observability.openshift.io/v1alpha1 kind: UIPlugin metadata: name: logging spec: type: Logging logging: lokiStack: name: logging-loki 

Step 5 — LogQL you will actually use in an incident

Query from the CLI against the gateway route, using your own OpenShift token:

LOKI="https://$(oc get route logging-loki -n openshift-logging -o jsonpath='{.spec.host}')" TOKEN=$(oc whoami -t) curl -sG -H "Authorization: Bearer ${TOKEN}" \ --data-urlencode 'query={kubernetes_namespace_name="payments-prod"} |= "panic"' \ --data-urlencode 'limit=50' --data-urlencode 'since=1h' \ "${LOKI}/api/logs/v1/application/loki/api/v1/query_range" \ | jq -r '.data.result[].values[][1]' 

Patterns worth memorising:

# 1. Narrow by label first, then grep. Always. {log_type="application", kubernetes_namespace_name="payments-prod"} |= "ERROR" != "retrying" # 2. Structured JSON logs, filter on a parsed field {kubernetes_namespace_name="payments-prod"} | json | level="error" | line_format "{{.msg}}" # 3. Which pod is burning? Error rate per pod over 5 minutes sum by (kubernetes_pod_name) ( rate({kubernetes_namespace_name="payments-prod"} |= "ERROR" [5m]) ) # 4. Slow requests from logfmt output {kubernetes_namespace_name="gateway"} | logfmt | duration > 2s # 5. HTTP 5xx extracted with the pattern parser, grouped by status sum by (status) ( count_over_time({kubernetes_container_name="nginx"} | pattern `<_> - - <_> "<method> <path> <_>" <status> <_>` | status =~ "5.." [5m]) ) # 6. p95 latency from an unwrapped numeric field quantile_over_time(0.95, {kubernetes_namespace_name="payments-prod"} | json | unwrap latency_ms [5m] ) by (kubernetes_pod_name) # 7. A component went silent — alert-grade query absent_over_time({kubernetes_namespace_name="payments-prod", kubernetes_container_name="worker"}[10m]) # 8. Who deleted the deployment? (audit tenant) {log_type="audit"} | json | objectRef_resource="deployments" | verb="delete" | line_format "{{.user_username}} -> {{.objectRef_namespace}}/{{.objectRef_name}}" 

Rule of thumb: every query must start with at least one label matcher that bounds the stream set, and a time range you actually need. {log_type="application"} |= "error" over 7 days will hit maxChunksPerQuery and time out. Narrow by namespace or container first.

Security best practices

  • No long-lived S3 keys. Use STS/IRSA on AWS (credentialMode: token), Workload Identity on GCP, Azure Workload Identity on ARO. If you must use static keys on ODF, rotate them and never commit them — the examples above build the secret from live cluster data, not from a file in Git.
  • Encrypt the bucket at rest with SSE-KMS and a customer-managed key, and block all public access. Logs contain tokens, user IDs and stack traces; treat the bucket as a crown-jewel data store.
  • Scope read access per tenant and per namespace. The gateway authorizes with SubjectAccessReview, so a namespace-scoped RoleBinding restricts a team to its own logs:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: lokistack-application-logs-reader rules: - apiGroups: ["loki.grafana.com"] resources: ["application"] resourceNames: ["logs"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: payments-logs-reader namespace: payments-prod subjects: - kind: Group apiGroup: rbac.authorization.k8s.io name: payments-team roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: lokistack-application-logs-reader 
  • Keep the audit tenant locked down. Only the security team should hold the audit reader ClusterRole, and audit retention should be driven by your compliance policy (often 30–365 days), not by the global default.
  • Redact secrets at collection time, not at query time. A drop or prune filter in the ClusterLogForwarder prevents the data from ever reaching the bucket — far better than relying on a reviewer not to look.
  • Do not expose the gateway route to the internet unless you need it. If you must, keep it behind the cluster ingress with OAuth and restrict source ranges; internal consumers should use the in-cluster service logging-loki-gateway-http.openshift-logging.svc:8080.
  • Apply NetworkPolicies in openshift-logging so only the collector and the gateway can reach the distributor and querier ports.

Decommissioning Elasticsearch

Run both stacks in parallel for one full retention period, confirm that your dashboards and alerts work against Loki, then remove the old stack. Delete the legacy ClusterLogging instance, uninstall the Elasticsearch Operator subscription and CSV, and only then delete the elasticsearch-* PVCs — that last step is the one that actually stops the invoice.

oc delete clusterlogging instance -n openshift-logging oc delete subscription elasticsearch-operator -n openshift-operators-redhat oc get pvc -n openshift-logging -l logging-cluster=elasticsearch # verify the list, then: oc delete pvc -n openshift-logging -l logging-cluster=elasticsearch 

Key takeaways

  • Elasticsearch is gone from OpenShift Logging 6.x; LokiStack plus Vector is the supported path, and it moves your log bytes from expensive always-on block storage to cheap object storage.
  • Size by daily ingest volume, not node count. 1x.small covers up to 500 GB/day; measure with loki_distributor_bytes_received_total before and after.
  • Retention is not on by default. Set spec.limits.global.retention.days, then use per-tenant and per-stream overrides to keep 30 days where it matters and 7 days everywhere else.
  • Ingestion limits are your cost and stability guardrail. When you see 429s, fix the noisy app or drop it in the forwarder before raising the limit.
  • LogQL rewards label-first thinking: bound the stream set, bound the time range, then filter and parse. Keep the eight queries above in a runbook.
  • Secure the bucket like a data store: STS credentials, SSE-KMS, no public access, per-tenant RBAC, and redaction at collection time.