Queue Execution Model

Queue Execution Model

How gondolier’s merge queue actually merges PRs: a stack of PRs is staged onto a queue branch, tested together once, and the result of that run is the gate that unlocks merging. This is the core contract of the system — everything else (provisioning, scheduling, wakeups) exists to make this flow reliable.

Restart survival is covered separately in queue-checkpoint.md.

The flow, end to end

PRs approved + queued            stage                      gate
─────────────────────────   ────────────────          ─────────────────────
merge_when_checks_succeed  shunt builds the           ci.yaml runs the full
= true (exact head sha)    mq/{base}/staging         suite ONCE on the
                           branch: cherry-pick        staged tree (mq/**)
                           each PR head onto it       trigger
                                   │                        │
                                   └────── RunStatus ───────┘
                                   (engine polls the workflow
                                    run by sha + branch)
                                          │
                                          ▼
                                        land
                                   ──────────────
                                   on success: shunt posts
                                   "merge-queue" = success on
                                   each PR head → branch
                                   protection check turns green
                                   → Forgejo auto-merge completes

Admission gate (shunt ≥ v0.13.0): before a scheduled PR is staged, the engine evaluates the base branch’s protection review policy (required approvals, block-on-rejected-reviews, stale-approval handling) against the PR’s reviews. A PR that can never satisfy it — approval blocked, live changes-requested — is refused at admission (no staging/CI run wasted) and a PR whose native merge times out twice on the same head is bounced instead of re-queued forever.

1. Stage

Shunt’s engine (shunt package, run inside the Go container) takes all PRs currently scheduled for auto-merge on the base branch and builds a batch:

1. Create the staging branch (mq/{base}/staging, e.g. mq/main/staging) from the base branch.
2. Cherry-pick each PR’s head commit onto it, in queue order.
3. Push the staging branch.

The APIStager (internal/gitops/stager.go) implements this with pure Forgejo API calls (create branch, create commit with the PR’s tree, update ref) — no local git. The batch is the entire stack: PRs are tested together, not individually.

2. Gate

The queue branch push triggers .forgejo/workflows/ci.yaml via its mq/** push trigger, which runs the full suite (vet, test, build, workers check, site check) on the combined tree.

Shunt’s engine polls the result with RunStatus(ctx, owner, repo, stagingSHA, stagingBranch) (internal/forge/forge.go), which queries Forgejo Actions for a workflow run matching the staging commit sha and branch, and returns its status (success, failure, cancelled, error, or still-running).

CI posts no statuses. The workflow run itself is the gate input. Shunt reads it; CI’s only job is to run the tests on the stack.

3. Land

When the run reports success, the engine posts a commit status on each PR’s head:

context:    "merge-queue"
state:      "success"
description: "merge queue: batch passed"

The branch protection (created during provisioning) requires the merge-queue context to be green on the PR head, so this status is what unlocks the merge — Forgejo’s scheduled auto-merge (merge_when_checks_succeed) then completes.

On failure, the engine posts merge-queue = error on the PR head (with the reason: head changed, batch failed, etc.) and the merge stays blocked.

4. Bisect

If a multi-PR batch fails the gate, the engine splits the batch in half and re-runs stage → gate on each candidate until the failing PR is isolated. A conflict during staging splits the batch at the conflict point so earlier PRs keep their place in the queue. This is how a bad PR gets identified without testing every PR individually.

Why this design

• Batch integrity: a stack of PRs must be mergeable together, not just individually. Testing them as one staged tree is the only way to catch cross-PR conflicts before anything lands.
• One test run: the queue branch is built once and tested once per batch — no N separate PR runs.
• The required check is the enforcement: because branch protection demands a green merge-queue context, no PR can merge by bypassing the queue (no direct merges, no manufactured statuses). The only way to satisfy the check is for shunt to post it after the batch passes.

Source of truth for the queue

The queue is forge-side auto-merge scheduling (merge_when_checks_succeed on the exact PR head). The engine enumerates open PRs and stages those still scheduled for auto-merge (queueEligibility → AutomergeState).

The LeaseDO keeps a local shuntState queue record (webhook _queueJoin/_queueLeave on PR opened/closed) — this is bookkeeping only; no code path feeds it to the engine. The container dispatch sends the token, repo, and base branch, and the engine reads the forge directly. Do not treat the DO queue state as authoritative.

Webhooks are wake signals: push/check/status events return wait_for_reconcile and the engine picks changes up on the next tick.

Status ownership

Actor	What it posts	Why
Shunt engine (tenant token)	merge-queue on PR heads (pending/success/error)	the gate result; the only status that unlocks merging
CI workflow	nothing	it only runs tests on the staged tree
Workers API	nothing gate-related	the API surface never touches gate statuses

Never manufacture a merge-queue status outside the engine. Doing so breaks the queue’s core guarantee (batch-tested merges) and the branch protection is there to make bypasses fail closed.

Trigger model

• Cron (correctness/recovery): a Worker cron trigger fires every 5 minutes (*/5). The scheduled() handler enumerates all managed repos and dispatches each to its LeaseDO.
• Webhook (optional, low-latency): when a repo’s wakeup_mode is webhook and gondolier registered a forge webhook, forge events POST to /api/v1/webhooks/forgejo, which triggers the same dispatch. Webhooks are never the sole source of truth — cron reconciles regardless.

Dispatch chain

1. scheduled() (index.mjs) → for each managed repo: LeaseDO POST /internal/reconcile with X-Container-Auth + the encrypted token.
2. LeaseDO (lease-do.mjs) dispatchReconcile: validates auth, acquires a lease, decrypts the token in-memory (envelope encryption; legacy btoa fallback), then calls SHUNT_CONTAINER (DO-style fetch to /dispatch).
3. ShuntContainer.fetch (shunt-container.mjs) routes the request to dispatch(), which calls the Go Container over the container binding (shunt_container.getTcpPort(8787) → POST /internal/reconcile).
4. The Go Container runs the standard Shunt engine (stage → gate → bisect → merge) and returns a safe summary (no credentials).
5. The dispatch writes the outcome to reconcile_status (ran_at, ok, generic error) — surfaced in the dashboard and the admin view.

Container wiring

Reconcile advances the queue by one step and is safe to call on a fixed interval — that’s the */5 cron and webhook wakeups described above. The container constructs the shunt engine as:

mq.Config{
  Owner, Repo, Base,
  StatusCtx:   "merge-queue",
  MergeStyle:  "merge",
  InstanceURL, PublicURL,
}

Workflows

• .forgejo/workflows/ci.yaml — PR/push CI and the mq/** gate (the whole run on the staged tree is the gate).
• .forgejo/workflows/deploy.yaml — production deploy, main only; never triggered by mq/** pushes.

Provisioning (best-effort)

On enrollment (and via POST /api/v1/user/repos/:id/provision or PATCH wakeup_mode), gondolier tries to configure the forge: register the webhook, protect the base branch, and add the repo’s status_context as a required check. Admin-gated actions need a repo-admin token; if unavailable the item reports needs_admin with manual instructions and enrollment succeeds regardless. State lives in managed_repos.settings; verification reads actual forge state so manual setup is also detected.

Operational flow for a repo

1. Enroll the repo + connection (token needs repo-admin for provisioning).
2. Provision: creates the webhook (webhook wakeup) and branch protection requiring the merge-queue status check on the base branch.
3. PRs: merge via the queue — queue with merge_when_checks_succeed=true, force_merge=false, and the exact PR head SHA.
4. Approval + CI green → the next reconcile tick (cron */5, or a webhook wake) stages the batch, runs the gate, and lands it.

Failure modes handled by the engine

Condition	Behavior
PR head changed while queued	re-queued, merge-queue = error on the head
Auto-merge cancelled	skipped, remainder re-queued
Gate failure (multi-PR batch)	bisect: split batch, re-stage/test
Gate run never appears	engine keeps polling (run still running)
Native merge times out	restore the queue entry, re-test the batch

Admin view

/api/v1/admin/* is gated by X-Admin-Key (constant-time, fail-closed when the ADMIN_KEY secret is absent). GET /api/v1/admin/repos lists all repos across tenants with config, owner (tenant id), lease state, and last reconcile; GET /api/v1/admin/stats returns totals. The admin page is gondolier.dev/admin.

Code map

• Engine: shunt/internal/engine/engine.go (stage/gate/land/bisect)
• Queue state machine + interfaces: shunt/mq/mq.go
• Staging: internal/gitops/stager.go (API-based cherry-pick)
• RunStatus: internal/forge/forge.go
• Container entry: internal/engine/engine.go, cmd/gondolier