We bring together a vertical audience that truly cares about experience—people who come from a review site, and who look not only at price, but also at stability, speed, and long-term usability. For site operators, this means you do not have to compete with others on the lowest price; quality and service can also bring you stable, high-quality traffic. For users, you can fairly see the true quality of each site, and, based on your own preferences and the dimensions you care about, choose a suitable API relay at a finer granularity. We currently have 3.7k GitHub stars, 1,000+ daily visits, 7,000+ average daily PV on the main site, and a community of 2,200+ people.

Folkbench is an independent evaluation platform for AI API relays. We do not evaluate the models themselves, but "how the same model performs on different API relays": whether calls are stable, how fast the responses are, how much you actually spend, whether your data is safe, and whether the model has been watered down or downgraded. Metrics such as availability and latency come from the platform's own continuous live measurements; prices come from each API relay's price API; data-security grades are manually verified by the operations team; and the rankings are synced every 15 minutes.

We have just completed a platform-wide upgrade. There are four main changes.

1. Rankings look only at measured results; payment does not affect ranking

Starting with this upgrade, all rankings are sorted only by measured data and data-security grade. What site operators pay for is monitoring, alerting, and promotion services. Payment does not buy a higher ranking, and it does not buy a higher data-security grade. Pinned ranking from the old partnership plans is being removed at the same time. We want every ranking you see to have only data behind it.

2. What question can each ranking help you answer?

  • Availability ranking: Can this route actually be used right now?
    This measures request success rate over the past 24 hours. Fake 200s, mid-stream disconnects in streaming output, and malformed responses all count as failures. They will not be passed off as requests that "look successful." Each route also includes a confidence interval calculated from the sample size. Routes with fewer samples have a wider interval, so you can see how reliable the number is.

  • Time-to-first-token ranking: Does it feel fast to use?
    This is the time from sending a request to receiving the first token, taken from live measurements over the past 24 hours. In tools such as Claude Code and Codex, it determines how long you have to wait before you see a reply.

  • Stability ranking: Will a long task drop halfway?
    This combines success rate, latency, and latency fluctuation into a stability score out of 100. The success rate uses a conservative value based on sample size, so a route that happens to succeed on just a few occasional requests will not be pushed to the front because of that. When you run agents, long tasks, and batch requests, look at this ranking first.

  • Session cost ranking: How much will it actually cost?
    This calculates the true cost of a typical programming session (2 million input tokens + 40,000 output tokens), and also gives an "API multiplier": the cost of the same session on the API relay divided by the official price. For example, 0.017 means 1.7% of the official price. You no longer have to convert each provider's multipliers, groups, and top-up exchange rates yourself.

  • Data-security grade: Will my code and conversations be retained?
    Grades run from 0 to 4, based on manual operational verification of three things: whether the operating entity can be verified; whether there is a public commitment on how user request content is handled and retained; and whether there is verification material that can be publicly cited. Expired evidence is labeled "Pending re-review." All four metric rankings are first ordered by data-security grade, and then, within the same grade, by each ranking's own metric.

  • Downgrade monitoring: Has the model been swapped out or downgraded?
    For the same model, every route answers the same unpublished set of questions in the same time window. The page shows each route's raw results side by side, including accuracy, input token count, needle-in-a-haystack over a long document, thinking tokens, and model fingerprint. We only display the numbers; we do not draw the conclusion for you.

  • API relay list and details: How is an API relay overall?
    Covered models, average availability, time-to-first-token, API multiplier, and data-security grade can all be seen on one page. The details page also has group prices and charts for each measured metric. You can also pick two API relays and compare them directly on the same model.

  • Switch client: Once you've chosen, how do you use it?
    A free, open-source desktop client that connects the API services you already have to programming tools such as Claude Code, Codex, and Gemini CLI, and lets you manage configuration and view local usage on your own machine. Keys are stored only on your own computer.

  • Evaluation methodology articles: Where does the data come from?
    We have published three model-evaluation methodology articles on the blog, covering 12 languages, explaining how each check is done and what its limitations are.

3. For users: Quick Check is available after signing in

Quick Check lets you use your own key to inspect an endpoint in about one minute, with optional items such as connectivity, model consistency, tool calling, and long context. To prevent abuse, it now requires you to sign in. Each account gets 5 uses per day, reset at midnight Beijing time. Sign-up supports email, Google, and GitHub, and we have added human verification and disposable-email blocking. Your key is not written into the link or browser storage.

4. For site operators: a new plan system; all payments are completed in the site-operator dashboard

  • Listing is free: site profile, models, and prices are displayed at no charge. Sites that have already joined but have not activated a plan keep their profile, model, and price listings; continuous probing is paused, and resumes after a plan is activated.
  • Basic plan $99/month: 10 monitoring items (one model in one group counts as one item), channel failure and recovery alerts, and review of security materials.
  • Premium plan $299/month: 50 monitoring items, stability alerts and webhooks, monthly and weekly data reports, on-demand probe API (100 calls included per month), README sponsor placement and Switch promotion benefits, and review of security materials.
  • Add-ons: extra monitoring slots $19/month (5 items); on-demand probe packs $19/100 calls and $149/1,000 calls (available with the Premium plan); README and Switch promotion can also be purchased separately, at $199/month each.

In the site-operator dashboard, you can fill in evaluation keys for each group yourself, view orders and invoices, renew, upgrade, and submit tickets. The bell in the upper right centrally shows alerts, ticket replies, reports, and purchase notifications.

The old "Professional Partnership" and "Premium Partnership" plans have been discontinued. Existing customers have already been moved to the corresponding new plans at no charge, with the same expiration date.

Site operators are welcome to join: https://folkbench.com/vendor-join

Next, our goal is to grow daily active users into the tens of thousands, and to become the most recognized evaluation and selection entry point in the API relay space. Want to see an evaluation of an API relay or a model that Folkbench does not cover yet? Please submit an evaluation request on GitHub, and we will arrange it as soon as we can.
https://github.com/Folkbench/folkbench