This job finished, and nobody will ever know. A customer asked for a 40-page report, and the model took six minutes. When it finished, your webhook endpoint was in the middle of a deploy and answered 503. The provider retried for about a minute, then gave up. An hour later, the output was deleted.
In this article we build long-running AI jobs that never lose a result: a webhook endpoint you can trust, a worker that applies each result exactly once, and a reconciler that catches what the webhooks miss.
Waiting for a slow model
A model call that takes minutes can't live inside a request. Hold the connection open, and something in between times out. Poll every two seconds, and you burn your rate limit on jobs that aren't done. So you give the provider a webhook URL, and it calls you when the job finishes.
But a webhook is a push with no guarantee. It can arrive twice, late, out of order, or never.
Process this webhook?
Four webhooks arrive for the same job. Would you process each one, or not? Decide before reading on.
-
succeeded, a newwebhook-id, a valid signature. - The same
webhook-id, 40 seconds later. - A valid signature, but a timestamp from two hours ago.
-
status=processing, arriving aftersucceeded.
One: process it.
Two: that's a retry. Ignore it, but still answer 200, or the retries keep coming.
Three: reject it. The signature is valid, but it's a replay.
Four: ignore it. A job never moves backwards.
Keep two, three and four in mind: the code below handles each of them.
The job table is the source of truth
Your API writes a job row first, then asks Replicate to run the model. The webhook endpoint only checks and queues: verify the signature, put the event on a queue, answer in milliseconds. A worker applies the result and copies the output to your own storage, because the provider deletes it after an hour.
And a reconciler runs every five minutes, asking the provider about any job that has been running too long. Webhooks are the fast path. The reconciler is the guarantee.
Following one job
Back to the job from the introduction. The API stores the job and creates the prediction. Six minutes later, the webhook hits the deploy: 503. The final retry fails too, and the provider gives up. Before, that was the end. Now, a few minutes later, the reconciler finds a job running too long, asks Replicate, and gets succeeded.
The worker copies the report to S3, and the customer gets it. Late, but not lost.
→ POST /reports job=4471 status=queued replicate.create ........ p_8f2k webhook=…/replicate 14:02:07 webhook ....... 503 (deploy in progress) 14:03:09 final retry ... 503 · provider gave up 14:05:00 reconciler .... job 4471 running 9 min → GET p_8f2k succeeded → copy output → s3://reports/4471.pdf job 4471 succeeded · 3 min late · 0 lost The webhook endpoint
The timestamp comes first: if it's older than five minutes, reject it. That's webhook three. Then the signature: an HMAC of the webhook ID, the timestamp and the raw body, compared with every signature in the header. Then queue the event, using the webhook ID as the queue's job ID.
A retry, like webhook two, lands on the same ID and adds nothing. And answer 200 right away: the provider expects a reply within a few seconds.
// webhooks/replicate.ts router.post('/replicate', raw({ type: '*/*' }), async (req, res) => { const id = req.header('webhook-id') ?? ''; const ts = Number(req.header('webhook-timestamp')); if (Math.abs(Date.now() / 1000 - ts) > 300) return res.sendStatus(400); const sig = req.header('webhook-signature') ?? ''; if (!verify(req.body, id, ts, sig)) return res.sendStatus(401); // Same webhook-id on every retry: the queue keeps one job per id await queue.add('apply', JSON.parse(String(req.body)), { jobId: id }); res.sendStatus(200); // in milliseconds: the worker does the slow part }); function verify(raw: Buffer, id: string, ts: number, header: string) { const expected = createHmac('sha256', SECRET) .update(`${id}.${ts}.${raw}`).digest('base64'); const sigs = header.split(' ').map((s) => s.split(',')[1]); return sigs.some((s) => safeEqual(s, expected)); } Apply once, never backwards
The worker applies the result in one transaction. Mark the webhook ID as seen; if it was already there, it's a retry, so stop. Then lock the job. If it's already finished, the event is late. That's webhook four: a job never moves backwards. If the prediction succeeded, copy the output to your own storage before it disappears, and finish the job.
The reconciler runs every five minutes and checks each job that is still running with the provider, through the same function. The webhook is a hint. The provider is the truth.
// jobs/apply.ts const TERMINAL = ['succeeded', 'failed', 'canceled']; export const apply = (eventId: string | null, p: Prediction) => db.tx(async (tx) => { if (eventId && !(await tx.markSeen(eventId))) return; // a retry const job = await tx.jobs.lock(p.id); // SELECT … FOR UPDATE if (TERMINAL.includes(job.status)) return; // late: never go back if (!TERMINAL.includes(p.status)) return; // still running const key = p.status === 'succeeded' ? await copyToS3(p.output, job.id) : null; await tx.jobs.finish(job.id, p.status, key); }); // Every 5 minutes: the webhook is a hint, the provider is the truth export async function reconcile() { for (const job of await db.jobs.runningFor('5 minutes')) { await apply(null, await replicate.get(job.predictionId)); } } Where this breaks
- Parsed bodies. Most frameworks parse JSON before your code runs, and the re-serialized body fails the signature check, which covers the raw bytes. Take the raw body on that route.
- Secret rotation. The header can carry more than one signature, so accept both the old and the new secret during the switch.
- Slow handlers. Download the output inside the endpoint, miss the deadline, and you get retried.
- The one-hour clock. The provider deletes outputs after an hour. Reconcile well inside it, and alert on any job older than that.
Targets to hold
- 0 lost results: every job ends finished, or someone gets an alert.
- 1× each result applied, however many times it is delivered.
- < 1 s to answer a webhook, so retries stay rare.
And nobody has to ask where their report is.
The numbers are targets for an illustrative scenario, not measurements.
▶️ Watch the video: https://www.youtube.com/watch?v=_Arkjh6IaZ4
💻 Full code: https://github.com/NodeNeuralAI/examples/tree/main/v6-webhooks
🔔 More enterprise AI architecture every week: https://www.youtube.com/@NodeNeuralAI?sub_confirmation=1


