Same Model Name, Different Answers: A Drift Probe for Free Model Endpoints

My eval suite passed on Monday. On Friday the same inputs failed. Nothing in my repo changed.

So what changed? The endpoint behind the model name.

I have been running small workloads against free model access, with the probe job itself living on a free server. This post is the FAQ I wish I had read first. It is a myth-busting list, and each myth comes with a check you can run yourself.

Why does the same request return different answers a week later?

Because a model name is a routing label, not a checksum. Providers patch weights, swap quantizations, and move traffic between backends. Your config file never hears about it.

You cannot prevent that. You can only notice it early, on your schedule instead of your users' schedule.

That is what the artifact below does. It fingerprints endpoint behaviour on a fixed prompt set, then compares runs.

Myth 1: "The model name in my config pins the weights"

This is the most expensive myth here. A name like some-model or some-model-latest is an alias. Aliases point at whatever the operator currently serves.

Even a dated snapshot name is a promise about weights, not about quantisation, batching, or serving hardware. Read your provider's versioning and deprecation docs before you assume pinning. I treat any pinning claim as unverified until I have seen drift logs.

This is where my probe workflow started. I run the probe against MonkeyCode's free model access, because free access is exactly where silent swaps are easiest to miss. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The corrected mental model: a model name identifies a contract, and only a probe tells you whether the contract still holds.

Myth 2: "temperature: 0 means deterministic"

No. Temperature zero removes sampling randomness, not implementation nondeterminism.

Floating-point addition is not associative. Batching and kernel selection change reduction order. Mixture-of-experts routing can vary with load. Thinking Machines documented this well in their write-up on defeating nondeterminism in LLM inference: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/

What does that mean for a probe? It means a single changed hash is a rumour. Two consecutive runs with the same change is a signal.

Build that tolerance into your gate before you build alerts. Otherwise you will page yourself about batching.

Myth 3: "A retry costs nothing"

Retries are cheap in your code and expensive in your ledger. A failed request may still consume tokens. A streamed request that dies halfway may bill for the prefix.

Now multiply by a retry loop with no cap. One flaky dependency becomes a token amplifier.

Before you blame the model, log four things per call: attempt number, status code, tokens reported, and wall time. If attempt three always succeeds with a partial token count, you have a retry policy problem, not a model problem.

Myth 4: "Any output change means the provider swapped models"

Sometimes the culprit is your own code. Check these before you open a support thread:

1. Did your system prompt or few-shot examples change?
2. Did you bump a client library that now sends different default parameters?
3. Did you change max_tokens, stop sequences, or a JSON schema?
4. Did you add a retry that silently drops a header?
5. Did your normalisation step change case, whitespace, or punctuation?

The probe isolates the model only if the probe itself is frozen. Pin the script. Commit the prompt list. Hash the request body, not just the response.

Myth 5: "A free server can host the probe forever"

Free servers are great at two things: running small cron jobs, and disappearing. Treat the host as replaceable from day one.

So do not keep your only history on that box. Write each run to local JSONL, then ship it somewhere durable. Even a git repo updated by a second cron job beats a lost disk.

Also keep the probe boring. No GPU, no framework, no build step. A 60-line standard-library script restarts anywhere in seconds.

The artifact: fingerprint.py

This runs on an OpenAI-compatible chat completions endpoint. It uses only the standard library, so the free server needs nothing but Python 3.

#!/usr/bin/env python3 """fingerprint.py - record endpoint behaviour on a frozen prompt set.""" import hashlib, json, os, time, urllib.request BASE = os.environ["LLM_BASE_URL"].rstrip("/") # e.g. https://host/v1 KEY = os.environ["LLM_API_KEY"] MODEL = os.environ["LLM_MODEL"] OUT = os.environ.get("FP_OUT", "fingerprints.jsonl") PROMPTS = [ "Reply with exactly: ok", "List the integers 1 to 5, comma separated, no spaces.", "What is 17 * 23? Reply with the number only.", "Complete with one word: 'red, green, ___'", "Write the word 'stable' five times, space separated.", ] def call(prompt): body = json.dumps({ "model": MODEL, "temperature": 0, "max_tokens": 64, "messages": [{"role": "user", "content": prompt}], }).encode() req = urllib.request.Request( BASE + "/chat/completions", data=body, headers={"Content-Type": "application/json", "Authorization": "Bearer " + KEY}, ) with urllib.request.urlopen(req, timeout=60) as r: return json.loads(r.read())["choices"][0]["message"]["content"] def norm(t): return " ".join(t.split()).lower() def main(): run = {"ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "model": MODEL, "base": BASE, "probes": []} for i, prompt in enumerate(PROMPTS): try: out = norm(call(prompt)) except Exception as e: out = "ERROR: " + type(e).__name__ run["probes"].append({ "i": i, "sha": hashlib.sha256(out.encode()).hexdigest()[:16], "out": out[:120], }) with open(OUT, "a") as f: f.write(json.dumps(run) + "\n") print(json.dumps(run, indent=2)) if __name__ == "__main__": main() 

Run it once by hand. Then run it again ten minutes later. If the hashes already disagree, you have measured nondeterminism, not drift.

compare.py: one file, no dependencies

#!/usr/bin/env python3 import collections, json, sys rows = [json.loads(line) for line in open(sys.argv[1])] by_model = collections.defaultdict(list) for row in rows: by_model[row["model"]].append(row) for model, runs in by_model.items(): first = {p["i"]: p["sha"] for p in runs[0]["probes"]} print(f"{model}: {len(runs)} runs") for run in runs[1:]: moved = [p["i"] for p in run["probes"] if first.get(p["i"]) != p["sha"]] print(f" {run['ts']} drifted={moved or 'none'}") 

Same idea, thirty lines. Now you have a baseline and a diff story.

Putting it on the free server

Two cron lines are enough. One records, one publishes history somewhere else.

# crontab -e 17 3 * * * cd /srv/probe && ./.env && python3 fingerprint.py >> probe.log 2>&1 25 3 * * * cd /srv/probe && git add -A && git commit -qm "probe $(date -u +%F)" && git push -q 

Before trusting it, restart the free server and confirm the job still runs. If boot wipes your directory, you just found out cheaply.

Reading the result: a triage table

Limitations and who should not use this

This probe detects change. It does not measure quality. A drifted hash can be an improvement.

Five short prompts are a smoke test, not an eval. If your product depends on structured output, pair this with a real schema validation gate.

Skip this approach if you need compliance-grade reproducibility. You need pinned weights, isolated hardware, and a vendor contract. A free endpoint cannot give you that, and neither can a cron job.

Also skip it if you have no place to store history. A probe with no baseline is just noise with extra steps.

What I would do on Monday

Run the probe three times against your current endpoint. Save the file. Then change nothing else for a week.

When the diff finally moves, you will know which side moved. That is the whole trick.

If you want to try it without a bill, a free model endpoint and a free server are enough to get the loop running. Keep the script boring, keep the history off the box, and never trust a single hash.