Originally published on Naveed Ahmed's Engineering Blog.
What if you could analyze Kubernetes failures and review Terraform configurations using an AI model running on your own hardware?
That's one of the most useful applications of local LLMs for DevOps.
Instead of sending every prompt to a hosted model API, you can run an open-weight model inside a controlled workstation or internal network.
Two tools worth understanding are Ollama and vLLM.
1. Why Run an LLM Locally?
Local inference can help teams:
- Keep more operational data within their own environment.
- Work without relying on an external inference service.
- Build internal tools for logs, infrastructure code, and runbooks.
- Control model selection, deployment, and inference infrastructure.
- Evaluate whether fixed infrastructure costs suit their workloads.
However, local deployment isn't automatically cheaper, faster, or more secure.
You still need to account for compute resources, model quality, access controls, and operational maintenance.
2. Ollama vs vLLM
| Feature | Ollama | vLLM |
|---|---|---|
| Primary use | Local development and simple inference services | Higher-throughput model serving |
| Setup | Straightforward | More infrastructure planning |
| Common environment | Laptop, workstation, private server | GPU-backed servers and Kubernetes |
| API | Local HTTP API | OpenAI-compatible serving API |
| Best starting point | Individual DevOps workflows | Shared team inference service |
Both can be useful, depending on the workload.
3. Run Ollama with Docker
For a CPU-based local test:
docker run -d \ --name ollama \ -p 127.0.0.1:11434:11434 \ -v ollama_data:/root/.ollama \ ollama/ollama Pull a coding-focused model:
docker exec ollama ollama pull qwen2.5-coder:7b Then query the local API:
curl http://127.0.0.1:11434/api/generate \ -H "Content-Type: application/json" \ -d '{ "model": "qwen2.5-coder:7b", "prompt": "Explain Kubernetes CrashLoopBackOff and suggest diagnostic commands.", "stream": false }' The model needs to be downloaded before offline use. The first pull requires network access unless the model files are already available locally.
For GPU acceleration, follow the Docker and Ollama documentation for your hardware.
4. Local Kubernetes Incident Triage
A useful starting point is a read-only assistant that summarizes Pod failures.
Collect the evidence:
kubectl describe pod my-app-123 -n staging kubectl logs my-app-123 \ -n staging \ --previous \ --tail=100 After checking the output for credentials or sensitive information, send relevant excerpts to your local model.
Ask it to identify:
- Likely failure causes
- Relevant log lines
- Missing evidence
- Commands to verify the hypothesis
- Recommended next steps
Treat the result as a troubleshooting suggestion, not a verified root cause.
5. Review Terraform Changes Privately
Another useful workflow is reviewing an infrastructure pull request.
For example:
terraform fmt -check terraform validate git diff -- '*.tf' A local LLM can help explain the changes or identify suspicious patterns.
But don't rely on an LLM as your only validator.
Use deterministic checks, infrastructure policies, security scanning, and human review before merging or applying changes.
6. When to Use vLLM
Ollama is a convenient starting point, but a platform team may need a centralized inference service for many applications.
vLLM supports high-throughput model serving and an OpenAI-compatible API.
For Kubernetes, consider:
- GPU node allocation and model memory
- Resource limits and scheduling
- Authenticated ingress or an internal API gateway
- Monitoring and request quotas
- Model versioning and persistence
- Network isolation and secrets handling
Do not expose an inference endpoint publicly without an appropriate security boundary.
7. Production Safeguards
Local inference still needs security controls.
Use read-only Kubernetes identities where practical, sanitize logs, authenticate model endpoints, and isolate inference workloads.
Review AI-generated changes before deploying them.
A useful first milestone is an assistant that explains failures and proposes checks, not one that autonomously modifies production.
Read the Complete Local LLM for DevOps Guide
My full article covers model selection, Ollama Docker setup, vLLM on Kubernetes, low-resource hardware, and practical SRE playbooks.
Read Local LLM for DevOps: Complete Guide
Final Thoughts
Local LLMs aren't a replacement for DevOps fundamentals.
They're another tool that can make investigation, documentation, and infrastructure review more convenient when deployed with appropriate controls.
Have you tried Ollama or vLLM for DevOps work?
Written by Naveed Ahmed — DevOps & Platform Engineering
