Skip to content

Escape divergence → table-move to custody

The problem

A table escape hands the pool to custody. Normally the game-server snapshot's per-player chip totals (Σ owed) equal the on-chain pool, so CustodyFacet.escapeToCustody(tableId, userIds[], amounts[], finalHashes[], …) debits the pool by Σ and writes one FREE claim per player.

But the engine/DB chip state and the on-chain pool can diverge — the pool is reset or drained out-of-band (a chain reset wipes tablePoolBalance) while the engine/DB retains the credits. Now Σ owed > pool, and the per-player escapeToCustody reverts on-chain with "Amounts exceed pool". The escape route surfaces a 500, the table stays funded, and the idle reaper's funded-abandoned path re-escapes it every sweep — the table is stranded in a retry loop and nobody's funds can be recovered. (Live: G2KP34, 2026-07-11.)

The decision

Read the actual on-chain pool inside the escape route, and branch:

resolve snapshot → claims[]           (Σ owed = Σ claims.amount)
read tablePoolBalance(tableId, token) = pool
─────────────────────────────────────────────────────────────
Σ owed ≤ pool   → per-player FREE claims        (unchanged path)
Σ owed > pool   → move the WHOLE real pool to ONE table-level hold
                  pool == 0 → off-chain 0-amount hold (ledger only, no tx)

The divergence branch calls escapeToCustody with a single synthetic entry whose amount is the entire real pool:

ts
escapeToCustody(
  tableIdHash,
  [tableHoldingId(tableId)],   // bytes16 = keccak256("hold:"+tableId)[0:16] — no real owner
  [pool],                      // the whole real pool — one entry, never overpays (pool ≤ pool)
  [tableFinalHash(finalHashes)], // keccak256 of the sorted per-player finalHashes
  token,
  handoffKey,
)

and writes one custody_entries row:

columnvalue
user_idnull — a table-move hold has no single owner
amountpool (exactly the on-chain pool)
origintable_move
stateneeds_reconciliation
idempotency_keymove:${tableId}
reconciliation_ledger[{ userId, owed, finalHash }, …] — the full per-player split

The per-player owed amounts and their position-chain finalHashes are not lost — they are preserved verbatim in reconciliation_ledger for a deferred operator reconciliation (Phase 2: split the hold into per-user claims against an operator-supplied shortfall allocation, conservation-guarded).

Why the players' JOURNAL SHARE as one entry (was: whole pool — revised by forced-freeze-cage-e2e D3)

The hold amount is pool − unreconciled rake (the rake summed from the poker.hand_players house rows — the journal), clamped at 0. The vault legitimately holds the players' money PLUS uncollected rake; moving the whole vault put the HOUSE's rake inside the PLAYERS' hold, and the game-server conservation identity then read −rake forever (the rake counted both in rakeTaken and inside movedToCustody) — the table could never close (live: C2UFJ4 −212, ZDA9W3 −8).

  • It can never overpay. pool − rake ≤ pool satisfies the contract's require(Σ amounts ≤ pool) by construction, exactly as the whole-pool move did — while also never claiming the house's money for the players.
  • The rake stays collectable. collect_rake / the rake sweep drains the retained rake through its normal path (cage-e2e T6) — nothing strands.
  • Degenerate cases preserved. No rake → the whole pool moves (the original shortfall semantics, T5/T10); pool ≤ rake → a 0-amount hold (ledger preserved, no tx).
  • No fairness decision is made at escape time. We only conserve — move the players' money somewhere recoverable. Deciding who gets what out of a shortfall is an operator/analysis step (Phase 2), not a hot-path guess.
  • The ledger is the audit record. Everyone's claim (owed) and memorialization anchor (finalHash) survive, so reconciliation re-anchors each per-user claim from its preserved finalHash.

Memorialization (no dangling chains)

Positions terminate at redeem (appendCustodyClaimEvent), not at escape — this is true for the per-player path too. The table-move hold captures every player's finalHash in the ledger, so each position's terminal hash is re-derivable at reconciliation → redeem. No chain is orphaned by the table-move.

Failure discipline (fund-path first-class)

FailureBehavior
Snapshot fault (game-server unreachable / non-200)503 escape_snapshot_unavailable; table stays frozen; nothing moved
Pool-read fault (tablePoolBalance revert/timeout)503 escape_pool_read_failed; table stays frozen; nothing moved
On-chain escapeToCustody revert500; table NOT closed; funds intact; retryable
Held entry claim attempt409 claim_needs_reconciliation (guarded before ownership)

Escape decision flow

Game-server reconstruction is conservation-clamped (forced-freeze-disposition)

The Σ owed the bank branches on comes from the game-server snapshot — per-player chip totals — via one of two paths: the live engine (pokerTable.escape(), a loaded room) or, when the engine is evicted/unloadable, a rebuild from the durable table_players (dbSnapshotFromRows). Normally the engine path is post-rake and correct — but a forced-freeze can corrupt either source (below), so both paths now pass through the same conservation clamp (the engine-path gap was found by this plan's falsification: Phase 2 clamped only the DB-truth rebuild, leaving a loaded corrupted engine able to over-move).

A forced-freeze (e.g. lease_lost mid-life) can evict the engine and leave those durable stacks at their PRE-rake (funded) value — Σ chip_balance == credited, not credited − rake. The reconstruction then reports a pre-rake Σ owed, the divergence branch moves the whole pool including the uncollected rake into the players' hold, and the game-server's boot conservation check diverges by exactly −rakeTaken (the rake counted twice: in rakeTaken AND inside movedToCustody) → escape 500 → the frozen-table reaper loops. (Live: C2UFJ4, 2026-07-28.)

The fix clamps the snapshot pool to the conservation ceiling at a single shared chokepointclampSnapshotToConservationCeiling in apps/game-server/src/shared/escape-conservation.ts — that BOTH snapshot paths (live-engine and DB-truth) pass through before the snapshot returns to the bank:

reconstructed pool = min(Σ chip_balance, credited − withdrawn − rakeTaken − existingCustody)

A healthy table's stacks already sit at (or below) the ceiling → the clamp is a NO-OP and every custody claim is paid in full — it is NOT a blanket rake-subtract (which would under-pay every healthy claim; staging proved 3/4 cash tables sit exactly at the ceiling). Only a corrupted, over-inflated stack sum is scaled DOWN (largest-remainder integer split → Σ == ceiling). Because BOTH snapshot paths run through the shared chokepoint, neither can hand the bank a pool over the ceiling — so a force-frozen rake-having table self-disposes cleanly (escape → custody → closed) instead of stranding in the reaper loop. sng / sandbox tables skip the clamp (the cash identity does not apply — mirrors checkChipConservation's scope).

The clamp fixes the game-server total handed to the bank. The on-chain vault table_move amount is now the players' journal share (pool − rake, forced-freeze-cage-e2e D3 — see above); the exact fair per-player split of a hold remains the deferred operator-reconciliation step.

New span admin.escape.reconstruct (admin.action) records escape.reconstruct.stacks_sum / .conservation_ceiling / .clamped_pool / .rake_taken / .clamped / .skip_clamp so a clamp is diagnosable AT the escape site, not only at the downstream boot conservation check.

The frozen-write guard — funded rows zero only with a journal entry (forced-freeze-cage-e2e D2)

Live forensics (ZDA9W3, 2026-07-29) exposed a second way a frozen table loses sight of its money: the per-seat cleanup paths race the table-level escape. After a lease-loss freeze, the agents' sockets drop → disconnect timers stand them seated → sitting_out (chips intact) → then a cleanup actor (system agent-leave / disposition sweep) persists gone/0 for every FUNDED row while its fund leg (forceWithdrawViaAdmin) fails silently on the frozen table — no custody or withdrawal journal rows. Both escape snapshot paths filter state != 'gone', so the escape then sees owed=0, misclassifies the table as the reverse divergence, and whole-pool-moves the vault (rake included).

The guard, at the ONE persistence chokepoint every cleanup actor funnels through (frozenDestructiveWriteAllowed in game-server participant-db.tsdbUpdateParticipantState / dbMarkLeft): on a frozen table, a destructive write (state→gone, balance→0) against a funded row is allowed only if a custody journal entry exists for it — its per-user escape claim or the table's table_move hold. Otherwise the write is refused loudly (participant.frozen_write_refused span + WARN) and the row stays visible to the escape — the only authorized mover of a frozen table's funds. Open tables and zero-balance cleanups are untouched (guard scope is frozen-only), and the post-escape settle passes because its journal entries exist by then.

Two falsification-hardened refinements (Phase 5, 2026-07-30):

  • Destructive means the write REMOVES value the row currently holds. The original shape-predicate fired on availableChips: 0 alone, refusing fund-PRESERVING transitions (a seated → sitting_out disconnect sweep with balances untouched) and churning a WARN per cycle on any long-frozen table (observed live, FF7T2K). A field already at 0 re-written as 0 is not destructive; state → gone always is (a gone row is invisible to the escape regardless of balances).
  • Only a VALUE-BEARING journal entry disarms. The table_move arm matches amount > 0 — a 0-amount hold (the pool≤rake degenerate's audit row) journals nothing and must not re-open the zeroing wound. Accepted limit: a partial-amount hold still disarms table-wide; the hold's ledger is the operator's map for reconciling exactly who was covered.

Telemetry

The existing custody.handoff span gains, on the divergence branch: custody.disposition='table_move', divergence.pool, divergence.owed_sum, divergence.shortfall. No new spans (no telemetry-contract change). The claim guard records custody.result='needs_reconciliation'.

Schema

Migration 0084_escape_divergence_hold:

  • custody_entries.reconciliation_ledger jsonb (null for normal per-player claims)
  • custody_origin gains table_move
  • custody_state gains needs_reconciliation + reconciled (the latter for Phase 2)

Solana parity (Phase 3)

Solana works identically, with one structural difference discovered mid-build: the Solana cage had no off-chain custody records at all — Solana custody lived purely on-chain as PDAs, so the whole off-chain surface (holds list, claim discovery, reconciliation) didn't exist for Solana. The fix reuses the same custody_entries table with a chain discriminator ('evm' | 'solana', migration 0085, mirrors walletAddress.chain); each bank's off-chain reads filter to its own chain.

  • numero-custody::split_custody_entry is the analog of the EVM facet fn, but splits one beneficiary per call (Anchor's fixed-account model — the cage loops the allocation). Same properties: no token move (vault untouched), per-call amount ≤ hold.amount (the on-chain conservation guarantee), consumes the hold at 0, admin(cage)-gated.
  • The solana-bank escape reads the real table-vault balance as the pool, and on divergence does a single whole-pool move_to_custody + writes the custody_entries hold row (chain='solana'). The reconcile route loops split_custody_entry + the shared reconcileHold. The same admin UI works on the Solana deploy (it hits BANK_URL; solana-bank exposes the same /admin/custody/holds + /admin/custody/reconcile/:entryId).
  • Proven end-to-end by a cage e2e (validator + all four programs + the spawned cage): divergent escape → whole-pool hold → conservation-refused bad split → pro-rata reconcile → per-user claim redeems on-chain.

Scope

  • Phase 1: divergence-safe escape on evm-bank + schema + claim guard. No contract change (reuses escapeToCustody + tablePoolBalance).
  • Phase 2: reconciliation endpoint + splitCustodyEntry facet + admin UI.
  • Phase 3: Solana parity (split_custody_entry + solana-bank escape/reconcile
    • the chain discriminator). Localnet-proven; devnet redeploy is the go-live step.

See .indusk/planning/escape-divergence-custody/ for the brief, test plan, and ADR.