A dealer uploads a CSV with 500 rows. Each row is a SIM card that has to be activated on one of several carrier APIs, and each activation costs real wholesale money that you cannot get back if the row was bad. That was the shape of the bulk activation feature I built in Laravel for a wireless dealer platform, and it is a good test of what Laravel bulk CSV processing with queues actually requires once money and third-party APIs are involved.
Most tutorials stop at "chunk the file and dispatch a job per row". That is the easy 20%. The hard 80% is deciding what happens before the first row is touched, who pays while the job runs, what a retry is allowed to repeat, and how the user finds out what happened. Below are the five decisions I made on that project, what each one prevented, and the one I would revisit today. If you are scoping a similar feature as part of an API integration project, these are the places where the estimate usually moves.
The client work is under NDA, so I describe patterns, not proprietary code. The full public write-up is in the bulk SIM activation case study.
Decision 1: Fail the whole file before anything is charged
Different carriers wanted different CSV shapes. Some needed an ICCID and ZIP code per row. One needed a ZIP code and a quantity, plus proof that inventory existed for that ZIP. The upload screen detects the shape from the header row, then runs a provider-specific preflight before any money moves.
The preflight for the ZIP-and-quantity carrier checks plan-code coverage for every ZIP up front. If one row cannot be fulfilled, the whole batch is rejected with a row-level error list. Nothing is held, nothing is sent.
I chose all-or-nothing on purpose. The alternative is to charge, call the carrier, find out it rejects row 212, and refund. That "charge then refund" path is where accounting bugs live: partial refunds, refunds that fail, and support tickets asking where the money went. A rejected file costs the dealer thirty seconds to fix. A wrong ledger costs hours.
The rule I took from it: validate against the upstream system's own constraints, not just your own. Your CSV validator knows a ZIP code has five digits. Only the carrier knows whether it has stock there.
final class BatchPreflight { /** @return list<array{row:int, error:string}> */ public function check(CarrierService $carrier, iterable $rows): array { $errors = []; foreach ($rows as $i => $row) { if ($problem = $carrier->preflight($row)) { $errors[] = ['row' => $i + 2, 'error' => $problem]; // +2: header + 1-based } } return $errors; } } This is a simplified sketch of the idea, not the client's code. The real preflight is provider-specific, and routing it through each carrier's service class keeps the controller from branching on carrier names.
Decision 2: Hold the money first, settle it row by row
Once preflight passes, the platform estimates the total cost of the batch and holds it up the wallet chain before the batch is saved. In this platform, a distributor can fund a dealer and a dealer can fund its own store, so one batch can touch more than one balance.
The hold happens in a single database transaction with the batch record. Two concurrent uploads from the same store cannot both pass a balance check, because the balance row is locked while it is read. In Laravel that is lockForUpdate(), which takes an InnoDB row lock; the MySQL locking reads documentation explains exactly what is locked and for how long.
DB::transaction(function () use ($store, $estimate, $rows) { $wallet = Wallet::whereKey($store->wallet_id)->lockForUpdate()->firstOrFail(); if ($wallet->available < $estimate) { throw new InsufficientFunds($estimate, $wallet->available); } $wallet->decrement('available', $estimate); $wallet->increment('held', $estimate); $batch = ActivationBatch::create(['store_id' => $store->id, 'held' => $estimate, 'total' => count($rows)]); $batch->rows()->createMany($rows); ProcessBatchJob::dispatch($batch->id)->afterCommit(); }); Note afterCommit(). Without it, a fast worker can pick up the job before the transaction commits and find an empty batch. That bug only appears under load, which is why it survives local testing.
As each row completes, the worker converts a slice of the hold into a real charge and releases the difference for failed rows. The tier that pays is decided per row: distributor-funded rows and self-funded store rows land on different ledgers. The point of holding up front is simple. Once the batch is accepted, the job can run for a long time and the dealer's balance is already accounted for.
Decision 3: One long job, with a deliberately long timeout
Each batch runs as a single queued job with a two-hour timeout. That sounds wrong if you have read "keep jobs short" advice, and in general that advice is right. Here it was a trade-off.
Some carrier sandboxes were slow and rate limited. A per-row job design would have needed a shared rate limiter across workers and a way to know when "the last row" finished. One job per batch kept ordering simple and made it easy to throttle, because one loop controls the pace of every call to that carrier.
| Approach | Good for | Cost |
|---|---|---|
| One job per batch | Strict carrier rate limits, simple ordering, easy "is it done" check | Long-running worker, a crash mid-batch needs resume logic |
One job per row (Bus::batch) | Parallelism, per-row retries, fine-grained progress | Needs shared rate limiting and a completion callback |
If I built it again for a carrier with generous limits, I would use Laravel job batches with a per-row job and a RateLimited middleware. The framework already gives you then, catch and finally callbacks plus progress counts, which is most of what I hand-built. For the slow, strict carriers on this project, the single long job was the more predictable choice.
Whichever you pick, set $timeout below the queue connection's retry_after, or the queue will hand the same batch to a second worker while the first is still running. That is the classic duplicate-activation bug, and with real SIMs it costs money. The queue documentation on timeouts and retry_after covers the relationship.
Decision 4: Make retries repeat only what failed
Carrier APIs fail in boring ways: timeouts, 502s, a token that expired mid-batch. The job records a status on every row (pending, activated, failed, with the carrier's message), and the dealer sees that list.
Retry is a separate action that re-submits only the failed rows. Rows already marked activated are never sent again. This is the same idea as an idempotency key, applied to your own data model. If the external API supports idempotency keys, use them as well; Stripe's idempotent requests are the reference design. Many carrier APIs do not, so your own row status is the only guard against activating the same SIM twice.
public function retryFailed(ActivationBatch $batch): void { $failed = $batch->rows()->where('status', 'failed')->count(); abort_if($failed === 0, 422, 'Nothing to retry.'); // re-hold only the cost of the failed rows, then re-queue $this->holdFor($batch, $failed); $batch->update(['status' => 'queued']); ProcessBatchJob::dispatch($batch->id)->afterCommit(); } Inside the job, the loop skips anything not in pending. A crash halfway through leaves a batch in an honest state: some rows activated, the rest pending, and a clear retry path. The same rule shows up in Stripe webhook handling: record the outcome before you acknowledge it, and make replays harmless.
Decision 5: Stream progress instead of polling for it
A dealer watching a 500-row batch does not want a spinner for several minutes, and I did not want 500 browsers each polling every second. The batch page opens a Server-Sent Events stream. The endpoint reads the batch record every two seconds and pushes completed, failed and total counters, then closes when the batch finishes or fails. It also closes after a five-minute cap, so a stream can never hang around indefinitely.
SSE is one-way, works over plain HTTP, and the browser's EventSource reconnects for you. See the MDN guide to using server-sent events for the wire format. For one-way progress updates it is far simpler than WebSockets; For the wider set of options, see my article on real-time features in Laravel SaaS.
return response()->stream(function () use ($batch) { $deadline = now()->addMinutes(5); while (now()->lt($deadline)) { $batch->refresh(); echo 'data: '.json_encode($batch->only('completed', 'failed', 'total', 'status'))."\n\n"; @ob_flush(); flush(); if (in_array($batch->status, ['done', 'failed'], true) || connection_aborted()) { break; } sleep(2); } }, 200, ['Content-Type' => 'text/event-stream', 'Cache-Control' => 'no-cache', 'X-Accel-Buffering' => 'no']); Two operational notes. X-Accel-Buffering: no stops nginx from buffering the stream until it closes. And every open stream occupies a PHP-FPM worker for its lifetime, so the five-minute cap is a safety valve, not a detail. If you expect many dealers watching at once, size the pool for it or move the stream to a dedicated process.
What I would change, and when not to build this
I would not hand-roll this for every CSV feature. A back-office import where a bad row costs nothing needs far less: validate, import in chunks, email a report. The hold, preflight and row ledger only earn their complexity when each row triggers an irreversible external action or spends money.
Things I would revisit on a new build:
- Move to
Bus::batchwith per-row jobs wherever the upstream rate limits allow it. - Store the upload and per-row results on durable storage from day one, so support can answer "what happened to row 212" without a developer.
- Alert on a batch that stops making progress, not only on failures. A job stuck inside a slow call produces no error at all.
Quick checklist before you start
- List every external call a row triggers and which of them cannot be undone.
- Write the preflight rules from the upstream system's constraints, then reject the file, not the row, when money is involved.
- Hold funds inside the same transaction that creates the batch, and dispatch with
afterCommit(). - Give every row its own status and make retries skip anything already done.
- Set
$timeoutbelowretry_afterand test with a worker killed mid-batch. - Decide how users see progress before you decide how the job runs.
If you are planning a bulk workflow like this and want a second opinion on the design, you can book a call and I will walk through the failure cases with you.
Originally published at aqibjavaid.site.
