Lease Management (LeaseDO)

Lease Management (LeaseDO)

Status: Authoritative lease protocol reference.

Cross-references: ARCHITECTURE.md (system overview), WEBHOOK_DESIGN.md (webhook dispatch), SECURITY.md (credential access during lease).

What LeaseDO does

LeaseDO is a Cloudflare Durable Object that provides per-(tenant, repo) concurrency control for queue operations. It ensures that only one worker execution processes events for a given repo at a time, preventing race conditions between concurrent webhook events and the cron reconciliation tick.

Data model

LeaseDO stores four entities in its durable storage:

lease

{
  "ts": 1722700000000,
  "by": "worker"
}

• ts: epoch milliseconds when the lease was acquired.
• by: identifier of the lease holder (always "worker" in current code).

queue

[
  {
    "prNumber": 42,
    "prSha": "abc123def456",
    "queuedAt": 1722700000000,
    "status": "queued" | "evaluating" | "gate_passed" | "bounced",
    "reevaluatedAt": 1722700100000,
    "gateResult": "success" | "failure",
    "gateResultAt": 1722700200000
  }
]

• prNumber: PR number on the forge.
• prSha: HEAD SHA of the PR at queue time.
• status: current queue state.
• reevaluatedAt: timestamp of last re-evaluation (new commit received).
• gateResult/gateResultAt: last gate outcome.

events

[
  {
    "ts": 1722700000000,
    "eventType": "pull_request",
    "action": "opened",
    "prNumber": 42,
    "details": null
  }
]

• Last 100 events only. Used for audit trail and debugging.

audit

[
  {
    "ts": 1722700000000,
    "action": "join" | "leave" | "reevaluate" | "gate_success" | "gate_failure" | "merge" | "bounce",
    "prNumber": 42,
    "prSha": "abc123"
  }
]

• Last 200 entries. Longer retention than events for support/debugging.

Lease primitives

acquire()

Claims the lease for this Durable Object.

1. Read `lease` from storage (may be null)
2. If lease exists:
   a. Check TTL: if (now - lease.ts) < 30s → 409 "locked", held_by = lease.by
   b. If (now - lease.ts) >= 30s → lease is stale, proceed to claim
3. Write `lease: {ts: now, by: "worker"}`
4. Return 200: {acquired: true}

TTL: 30 seconds. If the lease holder crashes or the worker dies mid-tick, the next caller sees a stale lease and claims it.

release()

Releases the lease.

1. Delete `lease` from storage
2. Return 200: {released: true}

Called in a finally block after every processEvent call, ensuring the lease is always released even on error.

check()

Returns current lease state without acquiring.

Response (200):
{
  "locked": true,
  "held_by": "worker",
  "expires_in": 15000
}

Response (200, no lease):
{
  "locked": false
}

Used by cron reconciliation and API callers to check lease status.

process_event protocol

The primary webhook dispatch path. This is the contract between the webhook handler and the lease manager:

POST /event
{
  "eventType": "pull_request",
  "tenant": "tenant_id",
  "repo": "owner/repo",
  "action": "opened",
  "prNumber": 42,
  "details": {"sha": "abc123"}
}

Flow:
  1. acquire()
     └─→ 409 if locked: return 409, event dropped, cron catches it
  2. _processEvent(eventType, tenant, repo, action, prNumber, details)
     └─→ Route to specific handler
     └─→ Write to events (last 100) and audit (last 200)
     └─→ Perform queue action (join/leave/reevaluate/gate_result)
  3. release()  (in finally block)
  4. Return 200 with result

Error handling: If processEvent fails between acquire and release, the finally block ensures release() is called. The event is lost — cron reconciliation picks up the inconsistency on the next tick.

Idempotency: _queueJoin checks for duplicate prNumber in the queue and returns already_queued if present. _queueLeave silently succeeds if the PR is not in the queue. These are the only idempotency guarantees — other events (duplicate check results, duplicate pushes) may produce duplicate actions.

Reconciliation endpoint

POST /reconcile
{
  "tenant": "tenant_id",
  "repo": "owner/repo"
}

Flow:
  1. acquire()
     └─→ 409 if locked by webhook event — skip
  2. _reconcile(tenant, repo)
     └─→ Check for stale lease (>90s)
     └─→ Check queue integrity (duplicates, bounced entries)
     └─→ Fix detected issues
  3. release()
  4. Return 200: {results: {fixed: [...], staleLeases: [...]}, queueLength, eventCount}

Reconciliation actions:

Detection	Action
Stale lease > 90s	Release stale lease, log
Duplicate PR in queue	Remove duplicates
Bounced PR still in queue	Remove bounced entries

TTL semantics and stale lease handling

Timestamp	TTL	Meaning
0-30s	Active	Lease is valid. Other callers get 409.
30s-90s	Stale	Next acquire() will overwrite. Lease holder crashed.
90s+	Dead	Cron reconciliation force-releases.

Why 30s TTL? Cloudflare Workers have a 30-second request timeout. The lease expires at the timeout boundary, ensuring a crashed worker’s lease doesn’t persist indefinitely. The 90s stale threshold gives cron three lease lifetimes before force-releasing.

Per-(tenant, repo) isolation

Each Durable Object instance is identified by a unique key derived from the (tenant_id, repo_slug) tuple. This is configured in the worker entry point:

// Worker routing
const DO_ID = env.LEASE_DO.idFromName(`${tenant}:${repo}`);
// ... dispatch to DO

Result: one Durable Object instance per managed repo. Each instance has its own storage namespace (ctx.storage), so there is zero cross-tenant leakage.

Edge cases

Double-staging prevention

If two webhook events arrive for the same PR (e.g., pull_request.opened delivered twice due to network retry), the _queueJoin handler checks for the PR’s presence in the queue before adding:

existing = queue.find(p => p.prNumber === prNumber)
if existing: return "already_queued"
queue.push(newEntry)

This ensures each PR appears at most once in the queue.

Timeout handling

The lease TTL (30s) is shorter than the Worker request timeout (30s) to ensure a crashed worker cannot hold the lease indefinitely. If a worker dies mid-tick:

1. Lease becomes stale (>30s).
2. Next acquire() call sees the stale lease and overwrites it.
3. Cron reconciliation detects stale state and force-releases.

Trade-off: A long-running event could be interrupted by a stale lease. The cron reconciliation picks up the missed event. This is acceptable because the cron runs every 5 minutes and corrects state.

Concurrent webhook + cron

When a webhook event is being processed and the cron tick fires:

1. Cron calls acquire() → 409 (webhook holds the lease).
2. Cron skips this repo, logs the skip.
3. Webhook finishes, releases lease.
4. Next cron tick acquires the lease and reconciles.

This is the expected behavior — cron is a safety net, not a parallel engine.

Queue state loss

If the Durable Object storage is corrupted (extremely rare), the queue state is re-derivable from the forge API:

• Cron reconciliation fetches PR state from forge API.
• PRs with auto-merge scheduled are re-queued.
• Closed PRs are removed.

This is a last-resort recovery path — the cron tick acts as a full state resync when the LeaseDO storage is unrecoverable.