Our referral program rewards the sponsor 16 days after the referee's payment. A cron takes that decision every morning. The referee's app has to be still active, its invoice paid without incident and without a credit note, and a few other checks have to pass. If a condition fails, the cron tries again the next day, and gives up 46 days after the payment.
Some context. I run backend engineering at GoodBarber, an app platform, and the referral program went live on October 23. Before launch, the review found three bugs in that decision. They have the same root. A check that runs 16 days later reads the database as it is now, and the database had changed in between.
Which date is "the payment"?
The first version counted the 16 days from the invoice date. An invoice can be created, go unpaid, and be settled later. When we regularize an unpaid invoice, only its settlement date changes. Counted from the invoice, the delay could end only a few days after the money actually arrived.
The delay and the 30-day retry window now start from the settlement date.
The row that was overwritten
One of the checks looks at the details of the referee's payment. It read the app's current direct-debit mandate, and updating bank details overwrites that row. If the details changed between the payment and day 16, the check looked at values the payment never used.
The payment history had the answer. Each direct-debit payment keeps the account it was taken from, next to its invoice. The check now reads that history as well as the current mandate. We tested it on an app that has a payment history and no current mandate.
"Already granted", checked by existence
Past a number of validated referrals, the sponsor earns a discount on a renewal. The cron grants it as a personal voucher with a pending use, which our renewal process attaches to the next renewal order. Before granting it, the cron checks whether that voucher already exists, so that it never grants it twice.
A chain of existing code broke that check. The renewal process attaches the pending use to an unpaid renewal order. If the customer changes plan before paying, the plan change deletes that unpaid order, and the order's delete hook deletes the voucher's use with it, but not the voucher. On its next pass, the cron finds the voucher and does nothing. The reward is gone, and nothing says so.
The fix leaves the existing billing code alone. A daily pass looks for tier vouchers that aren't deleted and have no use at all, pending or consumed, and gives them a pending use again, under the same per-app lock. To withdraw a reward on purpose, we mark the voucher deleted.
Running it again changes nothing
A decision that retries every day for 30 days runs many times on the same data. Each credit carries its invoice in its origin, referral_referee:<invoice> or referral_sponsor:<invoice>, and is only granted if that origin doesn't exist yet. The cron takes a PostgreSQL advisory lock per app, pg_advisory_xact_lock, and checks again once it holds the lock. During testing, we ran it several times in a row on a validated referral: it logged "validated" again and paid nothing. We never ran two passes at the same time.
The window also limits the kill switch
Commenting out the crontab line stops every reward without a deploy, while the page and the checkout keep working. The 46 days apply there too. A referral whose payment is more than 46 days old when the cron comes back is never rewarded.
What does your delayed job read: the state at the event, or the state when it runs?
