A job lease expires. Worker B takes over. Worker A wakes up with a result it computed before the handoff.

If completion is just UPDATE jobs ... WHERE job_id = ?, A can still overwrite B's result. The timeout made takeover possible. It did not stop A's code.

This example reproduces that sequence in SQLite, then makes the database refuse the old worker's completion. It includes an unsafe control and eleven local tests, with a controlled clock instead of sleeps.

The previous outbox example dealt with repeated delivery of one event. This is a different question: which worker's current claim may commit a result?

Give each claim a generation

A lease is temporary ownership of a job. A heartbeat renews it while the worker is still entitled to hold it. Expiry lets another worker claim the job.

The failure sequence is short:

Fixture time Action Stored ownership
0 A claims report-1 A, generation 1, expiry 10
10 B claims the expired job B, generation 2, expiry 20
10 B commits its result Done, generation 2
10 A submits its delayed result Refused by the guarded path

All numbers are test values. The fixture does not wait ten real seconds or run a report service.

Each successful claim increments the job's stored generation. Renewal keeps the same generation. Completion checks the job ID, owner, generation, state and deadline in its write predicate:

UPDATE jobs SET state = 'done', result = ?, owner = NULL, lease_until = NULL WHERE job_id = ? AND owner = ? AND generation = ? AND state = 'ready' AND lease_until > ?; 

Zero matched rows means refusal. It is not a successful completion with an empty result.

The generation matters even if the owner string is reused. A restarted process called worker-a can acquire generation 2; that must not revive a delayed operation carrying worker-a, generation 1.

The database participates in the decision

Martin Kleppmann's explanation of fencing tokens describes the general requirement: the protected resource must reject an older token. A check performed only by the worker can become stale before the write reaches its destination.

AWS's discussion of leader election also calls out pauses between checking a lease and performing work. It discusses using separate correctness mechanisms when more than one process believes it is leader.

This fixture uses a narrower arrangement: the lease row, protected job result and completion record live in one SQLite database. BEGIN IMMEDIATE acquires a write transaction before sampling the fixture clock. The guarded result update and the completion record commit together.

SQLite permits one simultaneous write transaction. Another claim therefore cannot commit between that guarded update and its commit. No transaction remains open while a worker performs the modeled computation; only claim, renewal and completion touch the database.

That is not a generic implementation of a distributed lock service. It is a local demonstration of keeping the ownership check at the same storage boundary as the protected change.

Test the worker that comes back

A takeover test that only checks “B eventually ran” misses the dangerous part. Retain A's old lease and actually attempt its completion.

The tests below check both orders: A returns while B still owns the job, and A returns after B has committed. In both cases, refusal leaves the current row unchanged.

Test Required observation
Unsafe completion after takeover A overwrites B's result, demonstrating the bug
Stale completion during B's lease Refusal; B's ownership and job data stay unchanged
Stale completion after B commits Refusal; B's result remains
Claim at the exact expiry boundary Before 10, B cannot claim; at 10, it can
Expired owner, with no takeover Completion is refused at the sampled deadline
Expired heartbeat It cannot revive the claim
Same owner name, newer generation The old generation is still refused
Altered job ID with another owner on that job Neither job changes
Two simultaneous claimers One gets a lease; the other gets none
Completion-record insertion fails The job update also rolls back
Live heartbeat It extends the current claim without incrementing its generation

The simultaneous-claim test releases two threads through a barrier. Each opens its own SQLite connection. The takeover cases use an explicitly advanced test clock; they do not depend on a scheduler pause happening by chance.

Run the complete example

Save this as worker_fence.py and run python worker_fence.py with Python 3.12 or later. It uses the standard library, creates temporary databases under test-db beside the file, and removes each test's own temporary subdirectory after closing its connections.

All eleven tests passed locally with Python 3.12.14 and SQLite 3.53.1.

"""Local lease-generation fixture. Python 3.12+; no queue or external API. Run: python worker_fence.py """ from concurrent.futures import ThreadPoolExecutor from contextlib import closing, contextmanager from dataclasses import dataclass, replace from pathlib import Path from threading import Barrier import sqlite3 import tempfile import unittest @dataclass(frozen=True) class Lease: job_id: str owner: str generation: int class LeaseLost(Exception): pass class Clock: """Trusted deterministic test clock, shared by the fixture's DB methods.""" def __init__(self, now=0): self.now = now def __call__(self): return self.now def connect(path): db = sqlite3.connect(path, timeout=5, autocommit=True) db.row_factory = sqlite3.Row return db def one(db, sql, args=()): with closing(db.execute(sql, args)) as cursor: return cursor.fetchone() @contextmanager def transaction(path): with closing(connect(path)) as db: try: db.execute("BEGIN IMMEDIATE").close() yield db db.execute("COMMIT").close() except BaseException: if db.in_transaction: db.execute("ROLLBACK").close() raise def initialize(path): with closing(connect(path)) as db: db.executescript(""" CREATE TABLE jobs ( job_id TEXT PRIMARY KEY, generation INTEGER NOT NULL DEFAULT 0, owner TEXT, lease_until INTEGER, state TEXT NOT NULL DEFAULT 'ready' CHECK (state IN ('ready', 'done')), result TEXT ); CREATE TABLE completions ( job_id TEXT PRIMARY KEY, generation INTEGER NOT NULL, result TEXT NOT NULL ); INSERT INTO jobs (job_id) VALUES ('report-1'), ('report-2'); """).close() def valid_ttl(ttl): if type(ttl) is not int or ttl not in range(1, 3601): raise ValueError("fixture TTL must be an integer in 1..3600") def claim(path, job_id, owner, clock, ttl=10): valid_ttl(ttl) if not isinstance(owner, str) or not owner: raise ValueError("owner must be non-empty") with transaction(path) as db: now = clock() # Sample after acquiring the DB write transaction. with closing(db.execute("""UPDATE jobs SET generation = generation + 1, owner = ?, lease_until = ? WHERE job_id = ? AND state = 'ready' AND (owner IS NULL OR lease_until <= ?)""", (owner, now + ttl, job_id, now))) as cursor: if cursor.rowcount != 1: return None generation = one(db, "SELECT generation FROM jobs WHERE job_id = ?", (job_id,))[0] return Lease(job_id, owner, generation) def renew(path, lease, clock, ttl=10): valid_ttl(ttl) with transaction(path) as db: now = clock() with closing(db.execute("""UPDATE jobs SET lease_until = ? WHERE job_id = ? AND owner = ? AND generation = ? AND state = 'ready' AND lease_until > ?""", (now + ttl, lease.job_id, lease.owner, lease.generation, now))) as cursor: if cursor.rowcount != 1: raise LeaseLost("renewal refused") def complete(path, lease, result, clock): with transaction(path) as db: now = clock() with closing(db.execute("""UPDATE jobs SET state = 'done', result = ?, owner = NULL, lease_until = NULL WHERE job_id = ? AND owner = ? AND generation = ? AND state = 'ready' AND lease_until > ?""", (result, lease.job_id, lease.owner, lease.generation, now))) as cursor: if cursor.rowcount != 1: raise LeaseLost("completion refused") # Both records are local to this same database transaction. db.execute("INSERT INTO completions VALUES (?, ?, ?)", (lease.job_id, lease.generation, result)).close() return result def complete_without_fence(path, lease, result): """Deliberately unsafe control; do not use as a completion path.""" with transaction(path) as db: db.execute("""UPDATE jobs SET state = 'done', result = ?, owner = NULL, lease_until = NULL WHERE job_id = ?""", (result, lease.job_id)).close() def snapshot(path, job_id='report-1'): with closing(connect(path)) as db: return dict(one(db, "SELECT * FROM jobs WHERE job_id = ?", (job_id,))) class FencingTests(unittest.TestCase): def setUp(self): base = Path(__file__).resolve().parent / 'test-db' base.mkdir(exist_ok=True) self.temp = tempfile.TemporaryDirectory(prefix='fence-', dir=base) folder = Path(self.temp.name).resolve() assert folder.is_relative_to(base.resolve()) self.addCleanup(self.temp.cleanup) self.path = folder / 'jobs.sqlite' self.clock = Clock() initialize(self.path) def claim(self, owner='worker-a', job_id='report-1'): return claim(self.path, job_id, owner, self.clock) def takeover(self, owner='worker-b'): first = self.claim() self.clock.now = 10 second = self.claim(owner) self.assertEqual((first.generation, second.generation), (1, 2)) return first, second def completions(self): with closing(connect(self.path)) as db: return one(db, 'SELECT count(*) FROM completions')[0] def test_unsafe_old_worker_overwrites_new_result(self): old, current = self.takeover() complete(self.path, current, 'new result', self.clock) complete_without_fence(self.path, old, 'stale result') self.assertEqual(snapshot(self.path)['result'], 'stale result') def test_stale_completion_cannot_disturb_new_lease(self): old, current = self.takeover() before = snapshot(self.path) with self.assertRaises(LeaseLost): complete(self.path, old, 'stale result', self.clock) self.assertEqual(snapshot(self.path), before) self.assertEqual(self.completions(), 0) complete(self.path, current, 'new result', self.clock) self.assertEqual(snapshot(self.path)['result'], 'new result') def test_stale_completion_after_new_commit_keeps_new_result(self): old, current = self.takeover() complete(self.path, current, 'new result', self.clock) before = snapshot(self.path) with self.assertRaises(LeaseLost): complete(self.path, old, 'stale result', self.clock) self.assertEqual(snapshot(self.path), before) self.assertEqual(self.completions(), 1) def test_live_lease_blocks_claim_until_exact_expiry(self): first = self.claim() self.clock.now = 9 self.assertIsNone(self.claim('worker-b')) self.clock.now = 10 second = self.claim('worker-b') self.assertEqual(second.generation, first.generation + 1) def test_expired_owner_cannot_complete_without_a_takeover(self): lease = self.claim() self.clock.now = 10 before = snapshot(self.path) with self.assertRaises(LeaseLost): complete(self.path, lease, 'late result', self.clock) self.assertEqual(snapshot(self.path), before) self.assertEqual(self.completions(), 0) def test_expired_heartbeat_cannot_revive_lease(self): lease = self.claim() self.clock.now = 10 before = snapshot(self.path) with self.assertRaises(LeaseLost): renew(self.path, lease, self.clock) self.assertEqual(snapshot(self.path), before) def test_same_owner_name_does_not_revive_old_generation(self): old, current = self.takeover('worker-a') self.assertEqual(old.owner, current.owner) before = snapshot(self.path) with self.assertRaises(LeaseLost): complete(self.path, old, 'stale result', self.clock) self.assertEqual(snapshot(self.path), before) complete(self.path, current, 'current result', self.clock) def test_wrong_job_is_refused_and_both_rows_stay_unchanged(self): first = self.claim() second = self.claim('worker-b', 'report-2') before = [snapshot(self.path, name) for name in ['report-1', 'report-2']] with self.assertRaises(LeaseLost): complete(self.path, replace(first, job_id=second.job_id), 'wrong job', self.clock) self.assertEqual([snapshot(self.path, name) for name in ['report-1', 'report-2']], before) self.assertEqual(self.completions(), 0) def test_two_simultaneous_claimers_get_one_lease(self): barrier = Barrier(2, timeout=5) def compete(owner): barrier.wait() return self.claim(owner) with ThreadPoolExecutor(max_workers=2) as pool: futures = [Source:https://dev.to/smtasker-official/a-lease-timeout-does-not-stop-a-worker-2m52

Related posts