I work on Folkbench. Teams often shortlist an AI API relay from price and uptime, then discover breakage only after Claude Code or Codex is already mid-task. Incomplete bodies, malformed tool calls, and identity drift under the same model label are different problems. Folkbench publishes three surfaces that map to those problems. This article is a practical order of operations for reading them on the live site.

The four questions (keep them separate)

  1. Availability — In a published window, how often did calls for this model × channel group succeed?
  2. Response completeness — When the transport looks fine, is the reply complete and shaped like a normal completion?
  3. Tools / JSON — Does the endpoint return tool calls and structured output your client can parse without hand-editing?
  4. Identity signals — In a shared probe window, do answer categories, token shape, and fingerprint fields look consistent with the model you asked for?

Mixing these into one “quality score” hides which layer failed. Fix the layer you can measure.

Surface A — Relay reviews: availability and published quotes

Open the English review page for the model family you actually call:

Workflow:

  1. Search the exact model ID your client config uses (version strings matter).
  2. Prefer the channel group you will bill, not a similarly named promo group.
  3. Read availability next to sample count and observation window.
  4. Compare input and output prices in the same currency; do not infer cache pricing from those two columns.

The on-page FAQs are operational: GPT access does not prove Codex client compatibility; one successful chat does not certify streaming, tool rounds, long outputs, or retries. Use reviews to build a shortlist of endpoints worth probing with your own key.

Surface B — API quality check: reproduce on your credentials

API quality check is the step you can re-run yourself.

  1. Paste API address, key, and model.
  2. Confirm you are authorized to test that key.
  3. Choose checks, then start. Folkbench issues the calls from its server and compares the returned payload.
  4. Read the private report. It is not written into public boards. Keys are excluded from reports.

Suggested check set for agent workloads:

  • Connectivity — short call; connection and reply shape.
  • Tool calling — tool protocol behavior.
  • JSON schema — structured output against a declared shape.
  • Long context or PDF reading when those appear in production.
  • Model consistency when you want repeated calls on identity and streaming stability.

Identity probe and fingerprint are always included, even if you leave other boxes unchecked. If this private run fails, stop before spending a long agent session diagnosing the client.

Surface C — Integrity Monitor: public same-window identity table

Integrity Monitor (Degradation monitor, v0.1) publishes same-slot identity probes across relays for one model. Typical fields:

  • Probe agreement (agree / total)
  • Input-token range
  • Reasoning-token median
  • Model fingerprint fields returned on the wire
  • Availability / first-token / session-cost when those monitoring columns are filled

Reading pattern:

  1. Filter to your model ID.
  2. Compare by group. Do not average every group at a station into one number.
  3. Look for disagreement among agreement rate, token shape, and fingerprint. Treat that as a cue to open Surface B on your key.
  4. Skip empty or pending probe cells; unpublished means unpublished.
  5. Note the page callout that some related model IDs share a base model and cannot be separated by fingerprint alone.

Public weekly rollups of integrity snapshots: weekly public snapshots. Short notes on how those rows are summarized: how to read weekly rows.

If you need a deeper engineering checklist for streaming escapes, multi-turn tool state, or cache-header pass-through beyond the form, use the software-engineering relay guide.

End-to-end check

  1. Shortlist on Claude relay reviews or GPT/Codex relay reviews: same model, group, and currency; write down availability, samples, window, and both prices.
  2. Scan that model on Integrity Monitor; note groups where identity fields look inconsistent with peer groups.
  3. Reproduce on API quality check with Connectivity + Tool calling + JSON schema (identity coverage is automatic).
  4. Smoke-test the real client: one streaming completion, one tool round, one longer generation. Keep the Folkbench private report next to the client log.

If a private check fails, inspect the failed check and returned error. If it passes but your client still fails, compare request headers, streaming events, tool schemas, and test conditions. Neither outcome by itself isolates the cause.

Why the order matters

Availability is cheap to skim and useful for excluding dead routes. Completeness and tools/JSON decide whether an agent loop can finish a turn. Identity signals decide whether you are comparing the model you think you bought. Running the private API quality check before a long coding session usually costs less than debugging a half-applied patch that came from a truncated or wrong-shaped response. Keep the three Folkbench surfaces bookmarked next to your client config so the loop above is muscle memory instead of a one-off audit.

Data limit

Public Folkbench tables can include display samples, and Integrity Monitor coverage is still uneven in v0.1. Treat published fields as monitoring signals for the stated window, not as proof that a model was replaced.

This article was edited with AI assistance; I checked every claim against the live Folkbench pages before publishing.