Queue Checkpoint Persistence

Queue Checkpoint Persistence

Status: Approved + implemented (staged). The design pivoted to shunt’s native Postgres-backed state store against Supabase (per operator direction), replacing the earlier LeaseDO-storage design. Implemented in a local branch (go.mod replace to the upstream fork); the upstream exposure lands in rbtr/shunt first.

Cross-references: queue-execution.md (the stage → gate → land contract), container-dispatch.md (the dispatch chain), PROD_READINESS.md.

Problem

The shunt engine keeps its queue state (the pending batch order and any active batch — a staged mq/{base}/staging branch waiting on its gate) in memory. It can persist that state through a CheckpointStore (LoadQueue / SaveQueue / DeleteQueue), but gondolier does not wire one, so a container restart mid-batch loses the in-process queue state. The engine then re-derives the queue from the forge and re-stages.

Today the in-process engine cache (one Engine per tenant/repo, rebuilt on container restart) covers the normal stage → gate → land cycle, and restart recovery is functional (validated: re-derive + re-stage + land). The gap is one of fidelity, not correctness: a restart can reorder the batch.

What the checkpoint actually preserves (verified in shunt v0.11.2)

applySnapshot restores pending (the queue order), the linger state, and the batch staging sequence; it sets active = nil — it does not restore the staged branch or its gate run. So recovery is always re-stage (the PRs return to the queue and a fresh staging branch is built); the checkpoint’s value is preserving the exact queue order and batch composition across restarts instead of re-deriving a numeric order from the forge.

Hard blocker: the checkpoint API is internal to shunt

The engine’s CheckpointStore interface and the snapshot types live in shunt/internal/engine and shunt/internal/checkpoint — Go internal packages, not importable from gondolier — and the public mq.Config exposes no checkpoint field. A gondolier-only implementation is therefore not possible today. Two prerequisite paths:

1. Upstream (recommended): add a public checkpoint field/API to rbtr/shunt’s mq package and export the snapshot types. Requires an upstream issue + PR + a version bump + the go.mod bump. Acceptance: the public API must expose Checkpoint on mq.Config and the QueueSnapshot/ActiveBatchSnapshot/PullRequestSnapshot types (or a marshal/unmarshal helper) with the same semantics as the internal CheckpointStore.
2. Vendor: vendor the shunt module and expose the field locally. Self-contained but adds a vendored dependency to maintain.

Until the API lands, the documented recovery model is re-derivation (current behavior, validated): restart → re-derive from the forge → re-stage → land.

Design (Supabase-backed — shunt’s native Postgres state store)

Shunt already ships a Postgres-backed queue state store (internal/checkpoint/postgres: shunt_queue_state + shunt_queue_leases tables, LoadQueue/SaveQueue/DeleteQueue and AcquireLease for replica coordination). gondolier runs on Supabase, which IS Postgres — so the checkpoint is simply shunt’s store pointed at Supabase, rather than a custom host-side store.

        Supabase Postgres (pooler :6543)
        shunt_queue_state_<tenant> / shunt_queue_leases_<tenant>
                          ▲
                          │ (pgx, simple protocol, one shared pool)
        container (Go) ───┘
        engine.Config.Checkpoint = postgres.Store (per tenant, keyed by
        owner/repo/base)   │
        engine.Config.Lease = postgres.Store (replica coordination)

• Store: the container opens the DSN (a container env var composed from worker secrets) with pgx + default_query_exec_mode=simple_protocol (the Supabase pooler is PgBouncer-style; prepared statements fail with SQLSTATE 42P05). ApplyMigrations creates the tables on first use.
• Tenant isolation: the upstream store takes a namespace (New(db, namespace)); the container keeps one store per tenant with a namespaced table suffix (shunt_queue_state_<tenant>), so tenants or forge instances sharing a repo identity never collide on the same rows. All stores share one connection pool (no per-tenant pools / idle connection growth).
• Wiring: engine.Config.Checkpoint (and, as a follow-up, Lease + LeaseHolderID + LeaseTTL) — the engine persists the queue snapshot after each reconcile and restores it on startup. The engine’s logger is threaded through mq.Config.Logger into a per-queue buffer, so each reconcile response’s log_tail carries the internal decisions (eligibility, staging, gate, land) for that queue only — reconciles for different queues run in parallel (per-queue lock, no global serialization).
• Credential trade-off: the container holds the Supabase DB password (as a container env var via the class envVars). This is a deliberate trade against the earlier LeaseDO-storage design (which kept DB creds worker-side): the shunt-native store buys real durability, queryable state, and multi-replica leases. The DB role can be least-privileged (the state tables only).

Recovery semantics

Restart → the container connects → loadCheckpoint restores pending (the queue order) from Supabase → the batch re-stages in the original order → gate → land. applySnapshot re-queues (staged-branch resume is not performed); the store preserves order + composition. Stale/merged PRs are dropped by resolve().

Security

The snapshot carries PR numbers, staging branch names, and shas — no credentials. It lives in the tenant-namespaced queue-state tables in Supabase (the per-tenant table suffix is the isolation boundary; tenants never share rows) and is reachable only by the container (the DB credentials are container env vars, never worker-side). Reconcile responses carry only the request’s own log_tail (per-queue log buffers, serialized per queue) and generic error messages — no tokens, no raw engine errors.

Rollout

1. The queue-state + queue-lease tables are created by ApplyMigrations on first tenant use (self-managed; no gondolier migration needed).
2. SUPABASE_DB_PASSWORD + SUPABASE_DB_HOST become worker secrets; the class composes the DSN into the container env.
3. Ships together via one full CI deploy.
4. The upstream shunt exposure must land first + the go.mod version bump + the local replace dropped before the gondolier merge.

Validation

• Unit: the fork’s postgres store tests + the gondolier wiring (DSN config, simple-protocol requirement).
• Integration (validated against the real Supabase pooler): the queue-state table persists the active batch; a simulated container restart (new process, same DSN) loads it and re-stages in the original order; the PR lands.
• E2E: the validated paths (single PR, multi-PR batch, gate failure, bisect, conflict) continue to pass.

Open items

• Lease wiring (follow-up): wire engine.Config.Lease to the postgres store (AcquireLease) for multi-replica coordination; currently only the checkpoint is wired (the DO lease still serializes dispatch).
• Least-privilege DB role: the container’s DB role should be scoped to the state tables.
• Upstream (owned): the public postgres store exposure lands in rbtr/shunt (the operator owns it); the gondolier go.mod bumps the version + drops the local replace.