Our stock feed service is a Java application that sends updates to a marketplace partner over TLS. The partner's endpoint uses a certificate from their own private CA, so the image our platform team builds has a small entrypoint script. The Dockerfile sets ENTRYPOINT to that script and CMD to java -jar /app/feed.jar. The script copies the JVM's default truststore, imports the partner CA from a mounted Secret with keytool, exports JAVA_TOOL_OPTIONS pointing the JVM at the new truststore, and then runs exec "$@", which starts whatever CMD says.
In October the service ran out of memory once and we had no heap dump. I added a command to the container in our Deployment: java with -XX:+HeapDumpOnOutOfMemoryError, a dump path on a volume, and -jar /app/feed.jar. It was the same command as before with two flags added.
The rollout completed and every pod went ready, because the readiness probe asks the service about itself. Every call to the partner failed with PKIX path building failed: unable to find valid certification path to requested target. Stock updates queued for fifty minutes before the alert on the queue fired.
In Kubernetes, command replaces the image's ENTRYPOINT, and args replaces its CMD. Setting command alone also means the image's CMD is ignored. I had meant to change CMD. I replaced ENTRYPOINT, so the script never ran, the truststore was never built, and the JVM started with the default one, which has never heard of our partner's CA. Nothing logged that the script was skipped, because the script was what would have logged.
We rolled back, and the change went in again as args instead of command, which keeps the entrypoint and passes it the new java line. The heap dump flags are still there. A CI check now reads each image's configuration with crane config and fails any manifest that sets command on a container whose image has an ENTRYPOINT, unless the pull request says why. The service checks at startup that its truststore contains the partner CA alias and exits with a clear message if it does not, so a pod without it never goes ready. The image's README, which nobody on our side had read, now opens with what the entrypoint does.
I was changing what the container runs. I did not know there was a step before it, and Kubernetes did not ask.
– Sergey Shinder
