Development & Operations

On this page

• 1. The async job system
• 2. pm2 processes (ecosystem.config.cjs)
• 3. The release process as it actually runs
• 4. The bug registry (artifact umbrella)
• 5. The Development section (being built)
• 6. Lineage today and the OpenLineage roadmap
• 7. Observability surfaces
• 8. Where to read release notes and reports

1. The async job system

1.1 Two queues, two workers

The generic queue lifecycle is queued → running → (completed | failed | cancelled) (scripts/job-worker.ts:16). The worker is stateless and restartable: on restart, running rows older than 5 minutes with no recent updated_at are reaped to failed (:24-26, reapStaleRunning, :617 exempts legitimately long kinds such as crawler.project_run).

1.2 Job kinds and handlers

Handlers live in one map, const HANDLERS: Record<string, JobHandler> at scripts/job-worker.ts:105 (the map closes at :430). Kind names are imported from config, never typed in the worker (headline rule) — with five string-literal exceptions (three crawler kinds and two dataset_analysis.* kinds). Measured: 16 config-keyed entries plus 5 literals = 21 kinds (grep -nE '^\s{2}(\[|")' scripts/job-worker.ts, lines 105-430; portal-to-ontology-development-writeup.md:102 counts 21 independently).

A handler receives (params, isCancelled, job). Cancellation is cooperative: the worker checks cancel_requested each poll and a handler halts at its next checkpoint (:18-22); crawler handlers refuse with cancelled_before_start if cancelled while queued (:324, :335, :363).

1.3 Lanes

config/job-lanes.config.ts declares named lanes; each runs at most one job at a time on its own DB connection, and the implicit default lane runs every kind not claimed (:10-17). Measured lanes (:36-49):

Enrolling a heavy kind is one config row, no worker edit (:16-17). The same file declares SYSTEM_ENQUEUED_JOB_KINDS (:80; rationale :54-79): kinds the platform enqueues with created_by IS NULL, which the job umbrella MUST NOT expose to a non-admin keyless caller (LIB-BUG-946 V-9).

1.4 Durable ticks

Time-gated schedulers (quality-axis refresh, recommendation refresh/attribution, RAG usage report) are gated on registry.scheduled_tick_state (scripts/job-worker.ts:43, :819-825) via armDurableTick(...) (defined :828; armed at :841, :960, :979, :991 — one per named tick). Policy lives in config/scheduled-ticks.config.ts: pollCadence: "PT5M", claimGrace: "PT10S", maxCatchUpWindows: 1, durabilityRequiredAtOrAbove: "PT30M" (ISO 8601 durations, :51-60); tick names at :88-92. Operators: a healthy boot logs durable-tick gate healthy (no degraded ticks); durable-tick DEGRADED means the schema is behind the code — migrate first (LIB-BUG-1029; CLAUDE.md release step 5).

1.5 The job umbrella (retry, cancel, correlation ids)

scripts/mcp-server/umbrellas/job.ts commands (measured case labels): get (:172), list_mine (:287), cancel (:334), retry (:396), history (:446), list_all (:504), stats (:533), reset_circuit_breaker (:552). Read-only get is open to any caller; listing/mutation of the analysis queue is admin-only (:168). retry is owner-scoped (asyncOwnerScope, LIB-BUG-428) and honours the failed/cancelled-only contract (:396-410); the analysis-queue status vocabulary is pending|running|completed|failed|retry|cancelled (:130).

Failures carry a correlation id. On handler failure the worker mints randomUUID() and stores {...err, correlation_id} in the row while logging failure correlation <id> for <short_id> (<code>) (scripts/job-worker.ts:588-596); reaped rows each get their own id (:656, :664). Operators: grep pm2 logs pdl-job-worker for the id a caller reports.

2. pm2 processes (ecosystem.config.cjs)

All processes run from the PRIMARY tree /home/datashield/library; worktree builds never deploy. DATABASE_URL is declared once (:13) and PDL_EXPECTED_DATABASE is derived from it so every process fails closed on a foreign current_database() (:14, LIB-BUG-719). Never pm2 reload --update-env (:12, :339, LIB-BUG-719). The three auxiliary workers are sourced from lib/workers/*.ts, not scripts/ (scripts/build-scripts.sh:84-86).

Which bundle each consumer needs, and the reload verb/order, is declared in config/bundle-consumers.config.ts (CONSUMER_RELOAD_POLICY) and consumed by npm run release:reloads (§3).

3. The release process as it actually runs

Authoritative text: CLAUDE.md § "canonical release-event sequence". TAG LAST. Summary of the sequence with the commands a runner types:

Builds: scripts/build-scripts.sh produces every dist-scripts/*.mjs (esbuild, type-stripped) and is REQUIRED before any bundle-consumer reload; env NODE_ENV=production npm run build produces .next/ for pdl-web (LIB-BUG-201 disaggregation, LIB-BUG-266).

3.1 Dev-gov rules a contributor MUST know

4. The bug registry (artifact umbrella)

Bugs are registry-backed rows in registry.bugs with a generated file; ids come only from the writer. Measured from the tool description in scripts/mcp-server/umbrellas/artifact.ts:

artifact.create({type:'bug'}) FORWARDS to bug_submit (:119); artifact.update/bulk_update/convert REJECT bug ids (:123-126). Directive 1 (docs/CC-DIRECTIVES.md:10) governs the bug file lifecycle.

5. The Development section (being built)

Sources: docs/audit/dev-section-charter.md §3-§4 and docs/audit/portal-to-ontology-development-writeup.md §1, §6. Status at v0.22.188: step 0 ratified in documents; no step-i code has been measured as shipped in this tree. Treat everything below as the plan of record.

5.1 Architecture (charter §3, dev-section-charter.md:28-46)

Development (nav section, config row)
 ├─ Monitor   — read models: Releases (build_cycles) · Bugs · Estate · Jobs        [step i]
 ├─ Pipelines — DEFINITIONS (registry.pipelines + append-only revisions)
 │             + RUNS (lineage.runs, one per node execution)                       [ii, iii-a, iv]
 └─ Lineage   — in-house canvas, third VIEW_SOURCE `openlineage_runs`, one pipeline run  [iii-b]

Rules (RFC 2119, charter §3): definitions reference catalog assets by id and MUST NOT carry a value (scanForValueLeak); the run ledger MUST be the OpenLineage store — no second run table; the executor MUST be one job kind pipeline.run on the existing queue (§1), cadence via the durable tick; every surface follows STANDARD-008.

5.2 Step i — the slice being built (PLANNED; writeup §6 rows i-a…i-f)

Step i total ≈ 455k naive tokens (≈ 680k at ×1.5) + vet ≈ 200k + runner ≈ 300k; zero migrations (charter §4).

5.3 Roadmap ii-iv (PLANNED — NOT SHIPPED)

Sequencing constraints (writeup §6): ii-a before iii-a; D-8 before ii-a; heal O-1..O-6 in code before ii-b; ii-b before iii-b; W1 before iv-a; iv-b before or with iv-a.

6. Lineage today and the OpenLineage roadmap

6.1 Today: the edges umbrella over asset_edges

scripts/mcp-server/umbrellas/edges.ts is read-only by contract — there is no edge-write command (:4, :51); relationships are written by the pipelines that own them (the corpus splitter writes part_of/cites_section). Commands (:55-92): predicates, neighbors (depth-1), traverse (bounded BFS, every hop gated), view (layered graph for a canvas), impact (reverse closure). Storage: migrations/registry/0190_asset_edges.sql; config/edges.config.ts:17 states the one rule — asset_edges stores ONLY otherwise-discarded relationships; everything else is read_through from its owning table (:42, :64). Measured state (writeup §1): lineage is real for RAG chunks (17,888 document→document edges) and absent for data — the lineage.* schema has 0 tables.

6.2 DESIGN: OpenLineage-native lineage (docs/specs/lineage-openlineage-design.md)

D1 canonical model Dataset / Job / Run (:62-101); D2 ONE run plane lineage.datasets / runs / io, relationship plane stays in asset_edges (:111); D3 emitter enrolment contract (:174); D4 wire: one read, one new command, no new tool (:228); D5 asset detail = three lists, no new tables (:270). Operator rulings 2026-09-16 (:346-367):

None of D1-D7 is implemented at v0.22.188 (lib/server/lineage/emit.ts absent; writeup §1).

7. Observability surfaces

The MCP surface uses the compact canonical MCP error envelope; problem+json (RFC 9457) is REST-only (DEC-1 / LIB-BUG-318; lib/server/problem.ts).

8. Where to read release notes and reports

Operator quick checks: pm2 list && curl -s localhost:3002/api/health; git status && git log --oneline -3; npm run release-lock:status; npm run release:reloads -- --plan; npm run release:migrate -- --plan.

Measured against the release named in the text. Raw Markdown: /ontology/documentation/development-operations.md. This page is also served as raw Markdown at the same URL with a .md suffix, for agents and search tools that prefer plain text. Every capability statement cites the source file and line it was measured from.