Table Ownership Lease

Date: 2026-07-28 · Status: shipped (N=1, enforcement unconditional — break-glass retired 2026-08-03) · Plan: table-ownership-lease

The problem

"Exactly one game-server process hosts a table" was an assumption, not an enforced invariant. During a deploy overlap (the old instance still draining while the new one boots and revives the same open tables) two processes could both advance the same hand and both sign payouts — the ~90k-chip duplication class (incident 3F6UUI). Nothing in the runtime prevented it; the environment shard key (tables.environment) only stops cross-environment double-hosting, not two instances of the same environment.

The decision

A per-table ownership lease on the turnstore Redis, acquired before a process constructs or revives a table, plus a fencing token that makes a stale owner's fund write unrepresentable. Scope is deliberately N=1: this plan does not add a router or multi- instance fan-out (those are named follow-ups — game-server-multi-instance, ws-gateway-fanout). At N=1 the lease is pure safety — it changes nothing about how the single instance behaves, and it is the structural floor the multi-instance work builds on.

Mechanism

• Lease key lease:table:{env}:{id} — SET NX PX (TTL 15s). Acquired in TableManager.loadOrCreate and at boot-revive, before the room is constructed. A refusal (owned_elsewhere) means another live process owns it → refuse to host rather than split-brain.
• Fencing token fence:table:{env}:{id} — a monotonic INCR per acquire, stamped on the room and persisted to tables.owner_fencing_token. BaseTable.persistAtBoundary refuses a boundary write whose token is not the current one (StaleOwnerBoundaryWrite), so even if two processes briefly overlap, only the latest owner's fund writes land.
• Renew heartbeat — every ~5s (TTL/3) the instance renews every lease it holds. A renew that finds the lease gone or foreign means ownership was lost.
• Freeze-loud on loss — a lost lease for a table this process is still hosting freezes it (transitionTableState(frozen, reason='lease_lost') → tryDealHand deals no more hands) and evicts the room (kills its timers, drops its connections). No live handoff — the winner rehydrates from the DB on its own boot. A lost renew for a table already released (idle eviction / graceful drain raced the heartbeat) is benign and does not freeze — only a still-hosted table can genuinely lose ownership mid-life (falsification T10).
• Break-glass — RETIRED (2026-08-03). The lease_enforcement game setting was the temporary rollout gate this section originally described; per its own "once proven, delete the read and the gate" clause it is now deleted — enforcement is unconditional (acquireTableLease(env, tableId) has no bypass parameter; there is no unleased hosting path). Evidence at retirement: the 48h staging soak (8+ real deploys, zero organic losses/refusals; every lease.lost an intentional smoke revoke) plus the live lease-loss disposition proof (FF7T2K — freeze-loud in 4s, refuse-to-dispose under a foreign owner, then frozen → escape → custody → closed, delta 0). NOTE: retired before the lease's first PROD exposure (prod receives it at the next pnpm release) — a deliberate deviation from U1's prod-soak-first sequencing, accepted because prod is the play-money beta and a prod misbehavior fails loud + recoverable (refuse-to-host / freeze), with revert as the hatch.

Same-owner takeover (the T5-soak finding)

The enforcement-ON soak surfaced a real regression: a restarted instance could not reclaim its own tables until the 15s TTL expired (owned_elsewhere at boot-revive), because the instance id carried the pid (which changes on restart) and acquire was NX-only. The fix, without weakening the split-brain guard:

• INSTANCE_ID is stable per host/machine — FLY_MACHINE_ID ?? FLY_ALLOC_ID ?? hostname(), no pid. There is exactly one game-server process per host (the N=1 invariant), so hostname is a safe unique id.
• Acquire is idempotent for the same owner — if the lease exists and is held by this owner, re-take it (refresh TTL, bump the fence). A different live owner is still refused. The bumped fence supersedes any in-flight write from the pre-restart process.
• On SIGTERM/SIGINT the instance releases every lease it holds, so a redeploy/restart frees its tables immediately instead of stalling the fresh instance for a full TTL.

Tradeoffs accepted

• A crash (SIGKILL, no graceful release) costs up to one TTL (≤15s) of rehydration delay for the affected tables — the TTL is the only live-vs-dead arbiter without a router. Graceful restarts (deploys) pay ~zero via the SIGTERM release.
• Two processes on one host would share an instance id and could both "own" a table — but the N=1 deployment runs exactly one process per host, and the fencing token is the fund-safety backstop even in a brief overlap. Multi-process-per-host is explicitly out of scope.
• Lease loss freezes rather than hands off. At N=1 a "lost" lease means Redis evicted the key or the process stalled past the TTL — an anomaly worth a human's attention (the U1 alert), not a silent live migration. Live handoff is the multi-instance plan's job.

Follow-ups

• game-server-multi-instance — a router that directs a table's traffic to its lease owner (turns N=1 refuse-to-host into N>1 correct-routing).
• ws-gateway-fanout — move the WS fan-out off the game process so ownership and connection handling decouple.

See the full ADR in the archive: .indusk/planning/archive/table-ownership-lease/adr.md. The lease-loss operational runbook: lease-loss.