DataShield Ontology: Overview and Architecture

On this page

• 1. What the Ontology is
• 2. Three surfaces, one declaration
• 3. The processing chain, end to end
• 4. Standards in force
• 5. Deployment shape
• 6. Versioning and release notes
• 7. Adoption checklist (normative)

1. What the Ontology is

The DataShield Ontology (production name still "DataShield MCP Dataset Library"; process prefix pdl-*) is an agentic data fabric: one server-side catalog of data assets, their governance state and their processing runs, exposed identically to AI agents (MCP), programs (REST) and humans (UI). It is not a notebook, a DAG scheduler, or a BI tool; those are consumers of the catalog, never peers of it.

The governing direction is the data catalog is the north star — docs/standards/STANDARD-011-catalog-north-star-DRAFT.md. Its headline rules, quoted as measured from the file:

Status caveat (measured): STANDARD-011 is DRAFT — not ratified; its §10 lists four operator decisions outstanding (STANDARD-011:3). Readers MUST treat R-1..R-20 as the direction the code is being moved toward, not as a description of what every code path does today. Where a stage below is absent or stub, the gap is stated.

Depth vocabulary used below (STANDARD-005 §13): real = measured working code with live rows; stub = declared seam that returns a typed pending result; absent = no code. Row counts are the read-only SELECT count(*) values recorded in CAP-MAP on 2026-09-22.

2. Three surfaces, one declaration

The platform serves the same capability set on three surfaces. The parity rule is STANDARD-011 R-14/R-15/R-16 (:153-171): settings, actions, parameters and help text MUST be declared once (config) and rendered by all three surfaces; a UI action MUST execute the same server-side capability as MCP; a surface MUST NOT introduce a control, default or permission the others lack.

Parity is enforced today only where a gate exists. npm run standards:parity-check (scripts/standards/parity-gate.ts, STANDARD-002 :18) gates the MCP-umbrella endpoint contract. The catalog-wide surface-parity gate R-14 names (audit:catalog-surface-parity) is PLANNED, not built (STANDARD-011:158-159). Adopters SHOULD therefore verify a capability on the surface they intend to use, not infer it from another surface. Not every umbrella has a REST twin: 18 REST bridge entries vs 45 umbrellas (measured ls); the data_catalog bridge is the reference implementation of parity-by-construction.

The MCP error envelope is not problem+json. MCP/JSON-RPC refusals use the compact canonical MCP envelope (config/mcp-rpc-codes.config.ts, imported at kernel.ts:11); REST uses RFC 9457 (CLAUDE.md headline; DEC-1 / LIB-BUG-318). The code extension is the same registry key on both surfaces (route.ts:14-15).

3. The processing chain, end to end

The Ontology has one processing model and it is catalog-shaped (CAP-MAP executive summary ¶1). One paragraph per stage; the "owning umbrella" is the MCP tool whose commands write the stage's tables.

3.1 Providers — owner data_catalog — real

A provider is a catalogued source system. Commands provider_create/list/get/update/retire/test (lib/data_catalog/dispatch.ts:433-486,:828). The kind registry PROVIDER_KINDS (config/data-catalog.config.ts:483, derived from PROVIDER_KIND_DECLARATIONS :227-482) declares each kind's STANDARD-010 connection path ("A" native dial, "B" REST+vault, or "ingest"). Table catalog.providers (migrations/registry/0220_data_catalog_schema.sql:41), 41 rows. UI app/app/data-catalog/sections/ProvidersSection.tsx. (CAP-MAP §A.1)

3.2 Connections — owner connection — real

Credentials plus the dial. Commands create/list/get/test/delete; dialers live in lib/data_catalog/resolver/dialer.ts (leased via withDialer :455). Path A dials natively for the seven measured kinds postgresql / snowflake / databricks / bigquery / sqlserver / mysql / s3; Path B saas_api dials via saas_get. Table registry.connections (0018_connections.sql:11), 1,341 rows. Secrets are vault-held (vault umbrella). Per STANDARD-010 there are two paths only; a provider-specific side door is forbidden (STANDARD-011 R-19 :191). (CAP-MAP §A.2)

3.3 Crawlers and scans — owner data_catalog (legacy crawler, crawler_run) — real

A crawler carries a purpose from CRAWLER_PURPOSES = discover | profile | extract | rescan (config/data-catalog.config.ts:910); data_catalog.scan_run (dispatch.ts:502-534) is the ONE admission door for reading provider data (R-4). A scan is recorded in catalog.scans (0220:143; `id, provider_id, purpose, status, scope, stats, error, started_at, finished_at, …), **98 rows**; crawler projects in registry.crawler_projects (0001_init.sql:57`), 213 rows. Execution runs as async-job kinds crawler.project_run / source_run / portal_discover (scripts/job-worker.ts:316,:334,:362) and the system-enqueued data_catalog.scan_run (config/job-lanes.config.ts:119-124). Lanes: harvest lib/data_catalog/harvest/apply.ts:320 applyHarvest(), profile profile/apply.ts:96 applyRelationProfile(), extract extract/index.ts:129 extractAssetToDataset() with a config-keyed EXTRACTORS registry (:47-77). The legacy crawler umbrella (13 commands) covers open-data portal crawls. UI app/app/crawlers/PageClient.tsx, app/app/data-catalog/sections/ScansSection.tsx. (CAP-MAP §A.3)

3.4 Assets, columns, terms, tags — owner data_catalog — real

Scans produce catalog.assets (0220:78, 4,549 rows), catalog.asset_columns (:108), catalog.terms (:206), catalog.mapping_decisions (:231) and catalog.asset_profiles. Commands `assets_list / asset_get / columns_list / columns_search / objects_search / profile_get / terms_materialize / terms_list / terms_grouped_list / asset_tag_set / asset_tag_remove / asset_register_enricher (dispatch.ts:95-96,:207,:535-671,:733`). terms_materialize is an async-job kind. Tags are free labels; classification labels are kernel outputs, not tags (STANDARD-011 R-8 :103-105; see §3.11). (CAP-MAP §A.4)

3.5 Bindings — owner data_catalog — real

A binding maps a catalog object to an MDM entity type — the MDM inbound leg. Commands bind_inbound / bindings_list / binding_get / unbind (dispatch.ts:671-701); rules in lib/data_catalog/bindings.ts:1-17 (full-extract requirement, primary-key ladder, STANDARD-006 P4 append-then-supersede, unbind deactivates the mapping in one transaction). Table catalog.bindings (0220:165), 27 rows; writes mdm.project_systems; governance-audit subject catalog_binding (migration 0225). Bind/unbind/remaster/terms_materialize are the four admin-floored verbs on the REST bridge (app/api/app/data_catalog/[verb]/route.ts:1-5). (CAP-MAP §A.5)

3.6 MDM / remaster — owners mdm (99 commands), mdm_project (15), stewardship (29) — real

data_catalog.remaster (dispatch.ts:701) decides run | skip | rescope on a pending remaster decision; lib/data_catalog/remaster.ts:1-13 hooks lib/conduit/autoDiff.ts onDatasetDiffed and enqueues mdm.resolution_run through the single ladder lib/mdm/resolutionEnqueue.ts. Resolution, training, rule enactment, preparer, recommendation and quality-axis work are async-job kinds (job-worker.ts:121-226). Golden records live in mdm.golden_entities (749 rows) with an outbox registry.golden_event_outbox (migration 0200). Stewardship UI app/app/stewardship/* (Sankey slots/FlowSankey.tsx) reads one flow_summary contract shared with data_catalog.flow_summary (dispatch.ts:706, lib/data_catalog/flowSummary.ts:1-12). (CAP-MAP §A.6, §A.15)

3.7 Datasets — owners dataset_catalog, dataset_ingest (9), dataset_admin (13), dataset_lifecycle (6), dataset_snapshot (7), dataset_data (2), file_repo (11) — real, with a stated disagreement

registry.datasets (0001_init.sql:102, 11,920 rows) is the platform's own asset plane: ingest (registry.ingests / ingest_steps 0045, ingest_jobs 0047), file storage conduit.repo_files (0017:99-123, carrying parent_file_id, derivation_type, derivation_config, pipeline_hash, version), snapshots, soft/hard delete and archive. REST /api/app/datasets/*, /api/app/my-datasets, /api/ingest; UI app/app/datasets/page.tsx, app/data-catalog/datasets. Disagreement (per META-CALIBRATION 5): STANDARD-011 N-4 calls for "one plane with namespaces", but today datasets are a second identity plane — ingest/publish/snapshot/transform outputs produce no catalog.assets row (CAP-MAP §A.7, W1 charter §0.1). A corpus IS a dataset (config/edges.config.ts:31). (CAP-MAP §A.7, §A.14)

3.8 Analysis engines (Conduit) — owner dataset_analysis (8) — real

Commands get / runs_list / create / submit / submit_batch / preflight / analysis_mode_get / analysis_mode_set; execution mode vocabulary client | server | auto (config/conduit-execution-mode.config.ts:37) with a parity contract between the server engines and the browser engine (:22-30). Server engines stamped by scripts/worker.ts: pg (:537), duckdb_profiler (:728), document_processor / document_disposition (:914,:929), worker_processor (:1016); large files fork scripts/worker-child.ts. The canonical 20-section analysis contract is lib/conduit/analysisContract.ts:244; deep analysis / RelationshipAtlas lib/analytics/deepAnalysis.ts:2. Runs in conduit.analysis_jobs (1,397 rows) and conduit.analysis_runs (0017:131); async kinds dataset_analysis.preflight / submit (job-worker.ts:378,:404). (CAP-MAP §A.8)

3.9 Transforms — owner dataset_transform (6) — real as compiler, absent as materialiser

Commands json_to_csv / json_process / transform_preview / transform_recipe_list / transform_recipe_get / transform_codegen (scripts/mcp-server/umbrellas/dataset_transform.ts:113-244). Model: 21 TRANSFORM_TYPES (config/transform-model.config.ts:55-77), 16 static recipes (config/transform-recipes.config.ts), compiler lib/conduit/transformModel.ts (compileTransformations :267, buildPreview :444, applyTransformations :526), pandas codegen lib/conduit/transformCodegen.ts (emitPandas :1048). Nothing is materialised: preview is in-memory and value-free; transform_codegen attach:true writes generated Python as a derivative file, not a dataset; config/edges.config.ts:266-268 states "dataset_transform materializes no datasets". aggregate / join / custom_sql / custom_python are refused with unsupported_reason. **Python is generated, never executed** on the server (P-3; the import-graph guard is declared as data in config/codegen-no-execution.config.ts (CODEGEN_NO_EXECUTION_SCAN) and asserted by the gated unit suite lib/conduit/__tests__/transformCodegen.test.ts, chained into release:gates at test:unit:gated — there is no audit:* script of its own). (CAP-MAP §A.9, ¶5)

3.10 Publish / archive / sync — owners dataset_lifecycle, dataset_admin — real as writes, absent as catalogued runs

dataset_lifecycle.delete / undelete / archive_create / archive_list / archive_delete / restore (lib/lifecycle/softDelete.ts, hardDelete.ts); dataset_admin.sync (:2650) → lib/sync/datasetSync.ts publishes to object-store / SQL targets with masking honoured (publish_masking_unapplicable :522); dataset_admin.catalog_export (:2646); join_create / union_create insert into conduit.repo_files directly. 34 write paths, 0 catalogued runs (W1 charter §1); the write seam catalogWrite() that R-5 requires is HELD. (CAP-MAP §A.10)

3.11 Classification and masking — owner classification (6), dataset_admin.masking_* — real (kernel in shadow)

STANDARD-009's pure kernel lib/classification/index.ts (`classifyColumn / classifyTable / deriveGovernance / classificationDigest) runs under config/classification-rollout.config.ts:6-11 (shadow DEFAULT / live / off`; the KERNEL_FORBIDDEN_IMPORTS gate keeps it pure). Masking is OFF by default and legally-required only (STANDARD-009 R2/R3): lib/query/maskingPolicy.ts, maskedRelation.ts, maskedMaterialize.ts; physical masking is an async-job kind (job-worker.ts:389). Tables registry.phi_determinations (0121:5), mdm.classification_systems / codes (0066:32,:69). unclassified is not public; readers fail closed (CLAUDE.md, STANDARD-009). (CAP-MAP §A.16)

3.12 Jobs — owner job (8) — real queue, absent DAG semantics

The ONE async queue is registry.async_jobs (0077_async_jobs.sql:29-59, 4,066 rows), consumed by scripts/job-worker.ts (HANDLERS :105, 21 kinds :109-404, dispatch :434, typed unknown_kind :436, failure shape {code,message,retryable,context} :496-533). Lanes are config-declared (config/job-lanes.config.ts:11-51), system-enqueued kinds at :67-137. Durable time-gated ticks use registry.scheduled_tick_state (migration 0227; lib/scheduler/durableTick.ts claimDurableTick :128). The older conduit.analysis_jobs worker (pdl-worker) runs beside it. Commands `get / list_mine / list_all / cancel / retry / history / stats / reset_circuit_breaker`. Measured gaps: no retry/backoff columns, no inter-job data passing, no fan-out/join, and no single enqueue seam — INSERT INTO registry.async_jobs appears in 17 non-test modules (CAP-MAP §A.11). Concurrency follows STANDARD-006 row-scoped patterns (P2 SKIP LOCKED etc.); table locks are forbidden.

3.13 Lineage — owner edges (5, read-only) — real for RAG containment, stub emitter, absent OpenLineage store

Commands predicates / neighbors / traverse / view / impact (scripts/mcp-server/umbrellas/edges.ts:55-92); predicate vocabulary config/edges.config.ts:79-172; WRITER_REGISTRY :196-215 declares three enabled writers: two RAG ones — writer:splitter and writer:citation_heal (lib/query/corpus.ts:935,:1143,:1179) — and writer:governance_harvest, the governance read lane's depends_on harvest (config/governance-control.config.ts:669, called at lib/data_catalog/harvest/apply.ts:488), which has written 0 rows to date. Store registry.asset_edges (0190:21-38), 17,888 rows, 100% document→document. REST /api/app/edges/view → lib/server/lineage/assembleView.ts (server-side longest-path layering); UI app/data-catalog/lineage/page.tsx (anchor-required :94-105) on components/lineage/LineageCanvasInner.tsx (@xyflow/react). Data lineage is not yet recorded: the emitter lib/data_catalog/extract/lineage.ts:109-116 emitLineage() returns {emitted:false, reason:"lineage_pending"} by type (:57), the lineage.* OpenLineage schema has 0 tables, and the seam R-10 names (lib/server/lineage/emit.ts) does not exist. Design: docs/specs/lineage-openlineage-design.md; gate audit:lineage-emitter-enrolment ships in warn. (CAP-MAP §A.12)

3.14 Workflows and RAG — owners workflow (4), rag (12) — real definitions; workflows are client-executed

workflow.list / get / run / save (umbrellas/workflow.ts:302-354) over 87 JSON recipes in library/workflows/*.json (schema lib/mcp/workflowRecipeSchema.ts, gate audit:workflow-recipe-integrity). **There is no server-side workflow executor** — the MCP client runs the steps (workflow.ts:324). rag indexes corpora (lib/query/corpus.ts) through the bulk job lane (job-worker.ts:257-305) and is the only writer that has actually written lineage edges — a third writer is declared and enabled but has produced no rows (§3.13). (CAP-MAP §A.13, §A.14)

3.15 Chain summary

4. Standards in force

All in docs/standards/. Status text is quoted from each file's header.

Adopters MUST also read docs/CC-DIRECTIVES.md (code-quality directives; Directive 9 adversarial vetting, Directive 11 privacy) and CLAUDE.md (headline value: no hardcoded values, configuration-based implementations, RFC/ISO standards).

5. Deployment shape

All processes run from the PRIMARY tree /home/datashield/library under pm2 (ecosystem.config.cjs, cwd: at :31,:67,:109,:152,:194,:222,:261,:293). Worktrees build but never deploy (STANDARD-007 :3-5). Bundles are esbuild ESM artifacts built by scripts/build-scripts.sh (:74-79) into dist-scripts/*.mjs; pdl-web is a Next.js build (npm run build, MUST run with NODE_ENV=production).

Database: PostgreSQL myorg_dataset_library with schemas registry, catalog, conduit, mdm (migrations under migrations/registry/). Health: curl -s localhost:3002/api/health. Release reloads are a derived chain (npm run release:reloads, config/bundle-consumers.config.ts CONSUMER_RELOAD_POLICY); hand restarts are forbidden (CLAUDE.md canonical release sequence step 6).

6. Versioning and release notes

• Version of record: package.json version (0.22.188 at this writing); pdl-mcp reads it at load

(scripts/mcp-server/kernel.ts:19-20 PKG_VERSION). The _hello umbrella returns version, current_release and available_versions (CLAUDE.md Rule 7a).

• Release snapshots: docs/versions/vX.Y.Z/ — 183 directories (ls | wc -l). Each holds RELEASE-NOTES.md

(front-matter fromVersion/toVersion/fromCommit/toCommit/manifestSchemaVersion/generatedAt), RELEASE-REPORT.html, capability-manifest.json (tools/commands/describe-coverage), describe-wire-reachability.json, diff.json and per-tool markdown under tools/ (measured at docs/versions/v0.22.188/). Snapshots are write-once.

• Release sequence: TAG LAST — lock → migrate → release:gates (205 chained links in this tree, derived from

package.json; CLAUDE.md's "174" is the count recorded at v0.22.170) → gen/snapshot/diff → publish → package-bump → tag → reload chain → T-8 report (CLAUDE.md, "canonical release-event sequence").

• Timestamps are RFC 3339 (generatedAt: 2026-09-21T03:10:43.420Z); durations/dates ISO 8601.
• Discovery for agents: ux.help and ux.docs_get accept a version parameter for multi-version documentation;

ux.versions_list enumerates the versions and takes no parameters (measured against docs/versions/v0.22.188/capability-manifest.json; CLAUDE.md Rule 7a names all three, which overstates the third).

7. Adoption checklist (normative)

1. Callers MUST enter provider reads through data_catalog.scan_run with a declared purpose (R-4); there is no other read door.
2. Callers MUST NOT expect dataset_transform to persist output or execute code — it previews and generates (§3.9).
3. Callers MUST NOT rely on data lineage today; only RAG document containment edges exist (§3.13).
4. Callers SHOULD treat the registry.datasets and catalog.assets planes as distinct until N-4 lands (§3.7).
5. REST callers MUST handle RFC 9457 problem+json refusals; MCP callers MUST handle the compact MCP envelope (§2).
6. Developers MUST extend an existing seam rather than add a second mechanism (R-20), and MUST route every change through

adversarial vetting before commit (CLAUDE.md headline; Directive 9).

1. Nothing in this document authorises building: the §8s hold in portal-migration-1-ratification.md — carried into

STANDARD-011's cross-reference table at STANDARD-011:295 — keeps catalog UI expansion on HOLD until layouts are approved by the operator.

Measured against the release named in the text. Raw Markdown: /ontology/documentation/overview-architecture.md. This page is also served as raw Markdown at the same URL with a .md suffix, for agents and search tools that prefer plain text. Every capability statement cites the source file and line it was measured from.