Running Ollama locally has become the simplest way into self-hosted LLMs: one command to install it, another to download a model, and an HTTP API ready to respond. It is one of the most followed open source projects on GitHub, used everywhere from a developer’s laptop to a team GPU server. As soon as an Ollama local LLM serves several people, it deserves to be monitored like any other service: metrics, logs and traces.
This simplicity has a downside. As long as Ollama remains a personal tool, nobody wonders how long a model takes to load or why an answer takes twelve seconds to arrive. As soon as it becomes a service that other applications depend on, it has to be treated like any other production component: you need to know whether it responds, how fast, with what GPU memory consumption, and for which callers.
This article introduces the project, installs it step by step, then shows how to monitor it: what the API exposes natively, how to collect the logs, how to produce OpenTelemetry traces on the client side and how all of this flows into an observability tool such as Dynatrace.
What is Ollama and what is it for?
Ollama is a language model runtime that wraps model download, quantisation and serving behind a command-line interface and a REST API. Internally it relies on llama.cpp and the GGUF format, which lets it run on CPU, on NVIDIA or AMD GPUs, and on Apple Silicon chips.
Its strengths for a technical team:
- a library of ready-to-use models (Llama, Mistral, Gemma, Qwen and many more), addressed by a name and a tag;
- a native API (
/api/generate,/api/chat,/api/tags…) and an OpenAI-compatible API under/v1, so existing SDKs can be reused; - automatic loading and unloading of models in memory.
Its limits are just as clear: Ollama aims for simplicity more than maximum throughput. To serve many concurrent users on datacentre GPUs, a server such as vLLM in production is often a better fit.
In practice: installing Ollama and serving a first model
Step 1 — Install the service
On Linux, the official script installs the binary and creates a systemd service:
curl -fsSL https://ollama.com/install.sh | sh systemctl status ollama On macOS and Windows, a graphical installer is available. In a container, the official image is used like this (add --gpus=all if the NVIDIA Container Toolkit is installed):
docker run -d --name ollama \ -v ollama:/root/.ollama \ -p 11434:11434 \ ollama/ollama Step 2 — Download and test a model
ollama pull llama3.2 ollama run llama3.2 "Explain what an SLO is in two sentences." ollama ps # models loaded in memory, CPU or GPU ollama list # models present on disk Available names and tags change constantly: check the model library on the project’s website before hard-coding a name in a script.
Step 3 — Call the API
By default, the server listens on port 11434 of the local machine. A non-streamed call returns, in addition to the text, timing information that is very useful for monitoring:
curl -s http://localhost:11434/api/generate -d '{ "model": "llama3.2", "prompt": "Why is the sky blue?", "stream": false }' | jq '{total_duration, load_duration, prompt_eval_count, eval_count, eval_duration}' Durations are expressed in nanoseconds. load_duration reveals the cost of a cold load; eval_count divided by eval_duration gives the generation throughput in tokens per second.
Step 4 — Open up the service with care
To expose Ollama to other machines, the OLLAMA_HOST variable sets the listening address. Other variables control how long a model stays in memory or the degree of parallelism. Their list and exact behaviour change from one version to the next: refer to the FAQ in the official Ollama documentation. The API has no built-in authentication: put a reverse proxy with access control in front of it, and never expose it directly on the internet.
How do you monitor an Ollama local LLM in production?
Monitoring a local LLM relies on four families of signals. Ollama provides only some of them natively; the rest is built around it.
Availability and model status
A simple call to /api/tags or to the server root is enough as an availability probe. The /api/ps endpoint lists the loaded models, their size and the share placed in GPU memory: it is the API equivalent of ollama ps, easy to query from a synthetic monitor.
Metrics
To our knowledge, Ollama does not expose a native Prometheus endpoint at the time of writing; check the release notes, as the topic is regularly discussed. Useful metrics are therefore calculated from API responses (total latency, load time, input and output tokens, throughput) and from infrastructure metrics. On the GPU side, NVIDIA’s DCGM exporter or a host agent provides card utilisation, memory and temperature.
Logs
Installed as a service, Ollama writes to the systemd journal; in a container, to standard output:
journalctl -u ollama -f docker logs -f ollama These logs show, among other things, GPU detection, model loading and memory errors. Verbose mode is enabled through a debug variable described in the documentation. A collection agent (OneAgent, or an OpenTelemetry Collector with the filelog or journald receiver) centralises them.
OpenTelemetry traces on the application side
The richest data lives with the caller. The OpenLLMetry project publishes OpenTelemetry instrumentations for several providers, including Ollama, which produce one span per call with the model, the tokens consumed and the duration, following the GenAI semantic conventions. Example in Python:
pip install ollama opentelemetry-sdk opentelemetry-exporter-otlp \ opentelemetry-instrumentation-ollama from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.instrumentation.ollama import OllamaInstrumentor import ollama provider = TracerProvider() provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) trace.set_tracer_provider(provider) OllamaInstrumentor().instrument() reponse = ollama.chat(model="llama3.2", messages=[{"role": "user", "content": "Hello"}]) print(reponse["message"]["content"]) Check the exact package and class names in the OpenLLMetry repository: these instrumentations change quickly. We compare this approach with Langfuse in our article on Langfuse and OpenLLMetry.
Sending Ollama telemetry to Dynatrace
The OTLP exporter is configured through the standard OpenTelemetry variables. For direct sending to Dynatrace SaaS, the environment’s OTLP endpoint and an API token with the appropriate ingest permissions (traces, metrics, logs) are enough:
export OTEL_SERVICE_NAME="assistant-interne" export OTEL_EXPORTER_OTLP_ENDPOINT="https://<environment>.live.dynatrace.com/api/v2/otlp" export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Api-Token%20<token>" export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" In a more serious deployment, it is better to go through an OpenTelemetry Collector, which receives traces and logs, adds host and Kubernetes attributes, then exports to the backend. The spans then appear in the distributed traces view, attached to the calling service, and the host running Ollama is correlated with its GPU metrics. The principle is the same as the one described in end-to-end OpenTelemetry: the value comes from the link between the model call, the service and the infrastructure.
Few dashboards are really useful: p50 and p95 latency per model, tokens generated per minute, share of calls with a cold load, GPU memory used, error rate. One alert on p95 latency and another on memory saturation cover the essentials.
Best practices
- Preload critical models and adjust how long they stay in memory to avoid cold loads at peak times.
- Never publish the Ollama port without authentication or network filtering.
- Instrument the caller with OpenTelemetry rather than inferring performance from server logs alone.
- Avoid recording full prompts and responses in traces if they contain personal data.
- Watch GPU memory: a model that spills over onto the CPU still works, but becomes much slower.
Frequently asked questions
Can Ollama replace a hosted LLM API?
For internal use, prototypes or sensitive data, yes, provided you accept smaller models and size the hardware appropriately. For heavy concurrent traffic, a throughput-oriented server is preferable.
Does Ollama expose Prometheus metrics?
Not natively, to our knowledge. You combine the durations returned by the API, the application’s OpenTelemetry traces and the host’s GPU metrics. Check the latest releases, as this may change.
How do you know whether a model is running on the GPU?
The ollama ps command or the /api/ps endpoint shows the CPU/GPU split for each loaded model; the startup logs also show the GPUs detected.
Conclusion
Ollama makes local inference almost trivial, but a shared LLM is still a production service. By combining API probes, centralised logs, GPU metrics and application-side OpenTelemetry traces, you get a complete picture, ready to be used in Dynatrace or another backend. To see where Ollama fits among the other building blocks, read our comparison of self-hosted open source AI stacks, and to add a chat interface to it, our Open WebUI guide.
span.observer is an independent community blog, not affiliated with Dynatrace LLC or the projects mentioned. Trademarks belong to their respective owners. Tools change quickly: check commands and options in their official documentation.
Originally published on span.observer. French version: https://span.observer/fr/ollama-llm-local-supervision/.
