Group Activity 2 — API Testing
Software Quality and Testing (SI-784) · Universidad Privada de Tacna
Team: Manuel Andre Dongo Palza & Milton H. Flores Chino
Project links
| 📦 Repository | https://github.com/miltom123/si784-grupal2-api-testing |
| 🚀 Live API | https://si784-api-envios.vercel.app |
| ⚙️ CI/CD runs (tests → Vercel → Newman & k6 in prod) | https://github.com/miltom123/si784-grupal2-api-testing/actions |
| 📝 Companion article (Newman, by Milton) | https://dev.to/milton_hfloreschino_123/pruebas-de-api-con-newman-de-0-a-121-aserciones-en-un-pipeline-que-despliega-en-vercel-2ke2 |
| 🎬 Video (≤ 5 min) | https://youtu.be/XAjGJB6H0Fw |
A 200 OK doesn't mean the result is correct
Picture an API that charges money. It returns 200, the JSON looks fine, the customer pays. But the total was off by a cent because, with twenty requests hitting at once, a rounding step ran on a half-written value. Single-threaded functional tests would never catch it: each one, on its own, returns 200. The bug only shows up under load — and in any system that moves money, a bug that only appears under load is the most expensive kind.
That's the gap neither unit nor functional tests cover, and it's what this article is about. In our project we split the testing of the same API into two fronts:
- Newman checks that each response is correct — schemas, contract, 121 assertions. Milton covers it in his article.
- k6 checks that the API stays correct and fast when everyone shows up at once. That's what follows.
One rule held across the whole project: nothing reaches production without passing, and the same tests run again against the public URL after deploying.
The API under test
A stateless REST API that quotes shipping prices between Peru's 25 departments (Node.js 20 + Hono, deployed on Vercel as a serverless function). The endpoint that matters isn't a health check that returns a constant — it's the one that does money math:
total = ((zone_base + S/2.50 × each_0.5kg + jungle_surcharge) × express_factor) + insurance_1% igv = 18% of the subtotal Base by zone, S/ 2.50 per started 0.5 kg, jungle surcharge, express factor ×1.6, 1% insurance on the declared value, and 18% VAT (IGV). Several chained roundings. Exactly the kind of logic worth stressing, because that's where a concurrency bug turns into real money.
The idea: a threshold is a contract, not decoration
k6 is Grafana Labs' load-testing tool. Scenarios written in JavaScript (versioned next to the code), a Go engine (hundreds of requests per second without choking the machine), and — the part that makes it fit CI — thresholds.
A threshold is an SLO written as code: "95% of quotes respond in under 800 ms", "fewer than 1% of responses carry an inconsistent amount". If it isn't met, k6 exits non-zero and the pipeline stops. The test stops being "numbers someone glances at" and becomes a quality gate that decides whether the change deserves production. It's the same logic as any serious quality control: it either meets the agreed limit, or it doesn't pass.
The whole scenario lives in one file: tests/k6/carga.js.
The full flow
flowchart LR A[git push to main] --> B[Local tests: unit + Newman + k6 smoke] B -->|green| C[vercel deploy --prod] C --> D[Production tests: Newman + k6 load] D -->|any threshold fails| E[Pipeline red: the change is not left deployed] D -->|all green| F[Trusted change in production] Scenario design: ramp up, hold, ramp down
Instead of dropping a fixed number of users all at once, the load climbs in stages. That way you see both the serverless cold start and the steady state:
export const options = { scenarios: { rampa: { executor: 'ramping-vus', startVUs: 0, stages: [ { duration: '15s', target: 10 }, // warm-up { duration: '30s', target: 20 }, // sustained load { duration: '10s', target: 0 } // ramp down ], gracefulRampDown: '5s' } }, thresholds: { http_req_failed: ['rate<0.01'], // < 1% HTTP errors http_req_duration: ['p(95)<800'], // 95% respond in < 800 ms tiempo_cotizacion: ['p(95)<800', 'avg<400'], // quote latency errores_de_negocio: ['rate<0.01'], // business errors checks: ['rate>0.99'] } }; Twenty virtual users isn't a big number; it's an honest one for this API. The goal wasn't to knock it over, but to set a realistic service level and confirm it holds, repeatably, on every push. And we measure by the 95th percentile, not the average: the average hides the users who waited too long; p(95) puts them front and center.
The star: a metric that watches the money
Here's what makes this test more than a plain "does it hold or not". k6 ships with network metrics out of the box (http_req_failed, http_req_duration). We added a business metric:
const erroresNegocio = new Rate('errores_de_negocio'); const tiempoCotizacion = new Trend('tiempo_cotizacion', true); // ... tiempoCotizacion.add(r.timings.duration); const ok = check(r, { 'quote 200': (res) => res.status === 200, 'total = subtotal + IGV': (res) => { const b = res.json(); return Math.abs(b.total - (b.desglose.subtotal + b.desglose.igv)) < 0.011; } }); erroresNegocio.add(!ok); // a 200 with a wrong total DOES count as a failure errores_de_negocio counts the responses that arrive with 200 but with amounts that don't add up. To the network that's a success; to whoever charges the customer, it's a failure. It's the only metric that catches the bug from the start of this article, and its threshold (rate<0.01) is as strict as the one for HTTP errors. We use closeTo with a one-cent tolerance because the total is born of roundings over decimals.
Two details that separate a serious test from a toy one
Random traffic, not the same case a thousand times. If you always repeat Tacna → Lima, 2 kg, the platform caches and you measure a lie. Each iteration builds a different shipment — weights from 0.1 to 50 kg, varied origins and destinations (including the jungle, which adds a surcharge), 30% express — so every branch of the calculation runs.
Expected errors aren't failures. One request sends a weight of 999 kg on purpose, to confirm validation still returns 422 under load. The catch: k6 counts any 4xx as a failure by default, which would push http_req_failed to 33%. So we declare it explicitly:
http.post(url, invalidBody, { responseCallback: http.expectedStatuses(422) }); As a bonus, setup() does a GET /v1/health before starting: if the API doesn't answer, the test aborts with a clear message instead of generating thousands of meaningless errors. And handleSummary() writes reports/k6-resumen.md, which GitHub Actions shows directly in the run summary.
k6 inside the pipeline
In ci-cd.yml, k6 runs twice, with different intents:
- uses: grafana/setup-k6-action@v1 # Before deploying: a quick smoke so we don't burn minutes if something is already broken - run: k6 run --vus 5 --duration 10s -e BASE_URL=http://localhost:3000 tests/k6/carga.js # After deploying: the full ramp scenario against production - run: k6 run -e BASE_URL="$BASE_URL" tests/k6/carga.js The short smoke cuts the obvious errors early; the full test, now against Vercel, validates the real service level.
Results against production (Vercel)
| Metric | Value |
|---|---|
| Requests | 3148 |
| Requests per second | 56.9 |
| HTTP errors | 0.00% |
| p(95) duration | 45 ms |
| Average quote | 26 ms |
| Checks passed | 100% |
All five thresholds green: http_req_failed < 1%, http_req_duration p(95) < 800 ms, tiempo_cotizacion p(95) < 800 ms and average < 400 ms, errores_de_negocio < 1%, and checks > 99%.
The number that made us think most: locally the p(95) sat around 2 ms; in production, 45 ms. The code didn't get worse — that's the network plus the serverless layer. That 20× gap is exactly why you have to measure where the code runs, not where it's written.
Why this matters more, not less
Today a growing share of code — and soon much of the tests themselves — is written by AI. That speeds everything up, but it shifts the question from "did I write it right?" to "how do I know what got generated behaves well in production?". The answer isn't reading more lines: it's having an executable quality gate that runs on its own with every change and says yes or no with data. A k6 threshold isn't impressed by elegant code; it only measures whether, under real load, responses arrive on time and the amounts add up. The faster we produce, the more that gate is worth.
Takeaways
- The threshold is the test. Without
thresholds, k6 just prints numbers; with them, load becomes a gate that blocks deploys, like any SLO. - Performance and correctness are measured together. A fast
200with a wrong total is still a bug — and only a business metric catches it. - Declare your expected errors, or your failure metrics will lie.
- Measure where it runs, not where it's written. The jump from 2 ms to 45 ms between local and production shows up in no test that doesn't hit the public URL.
Try it yourself:
k6 run -e BASE_URL=https://si784-api-envios.vercel.app tests/k6/carga.js And the correctness of each response one by one — schemas, contract, edge cases? That's Newman's job, and Milton explains it in his article.

