# 04 — Datasets, Analysis and Transforms

**Measured against:** `/home/datashield/library` at v0.22.188 (`package.json:3`), 2026-09-22.
**Audience:** developers, analysts, AI agents routing through the MCP surface.
**Method:** every claim below is read from code or migrations and cited `file:line`. Where the platform does
*not* do something, this document says so explicitly rather than implying capability.

---

## 1. The dataset model — `registry.datasets`

The dataset is the unit everything else attaches to. Its row was created in `migrations/registry/0001_init.sql:102-142`
and extended by later migrations. The base columns:

| Group | Columns (from `0001_init.sql:102-142`) |
|---|---|
| Identity | `id UUID PK`, `provider registry.provider_type NOT NULL`, `provider_dataset_id` |
| Description | `title`, `description`, `publisher_name`, `jurisdiction_level`, `jurisdiction_name`, `license`, `tags TEXT[]`, `topics TEXT[]` |
| Source | `landing_url NOT NULL`, `landing_url_canonical NOT NULL`, `raw_page_payload JSONB`, `raw_source_payload JSONB`, `steward_notes` |
| Liveness | `is_active`, `confidence`, `discovered_at`, `last_checked_at`, `last_changed_at`, `next_check_at`, `check_status`, `http_status`, `content_hash` |
| Usage | `read_count`, `search_hit_count`, `last_accessed_at`, `search_tsv` |
| Audit | `created_at`, `updated_at` |

Later migrations add **53** more columns by `ALTER TABLE registry.datasets ADD COLUMN` (measured across
`migrations/registry/*.sql`; none is ever dropped):

| Family | Columns |
|---|---|
| Profile / shape | `column_count`, `row_count`, `file_count`, `total_file_size_bytes`, `source_file_size_bytes`, `completeness_score`, `quality_score`, `quality_rating`, `has_analysis`, `table_type`, `dataset_type`, `file_type` |
| Governance | `compliance_level`, `has_pii`, `has_phi`, `classified_by`, `last_classified_at`, `needs_classification`, `visibility` |
| Provenance / capture | `source_category`, `source_system`, `source_mode`, `source_total_rows`, `row_count_drift`, `provider_metadata`, `capture_metadata`, `extras_json_text`, `notes_json_text`, `ai_tags`, `ai_tags_updated_at` |
| Lifecycle | `lifecycle_stage`, `deleted_at`, `deleted_by`, `purge_after`, `content_erased_at` |
| Query engine | `query_engine`, `query_engine_loaded_at`, `query_engine_last_refresh_at`, `query_store_size_bytes` |
| Refresh | `last_refresh_status`, `last_refresh_actor`, `last_refresh_attempted_at`, `last_refresh_completed_at`, `last_refresh_duration_ms`, `last_refresh_message`, `schedule_next_run_at` |
| Snapshot / archive | `snapshot_count`, `snapshot_total_size_bytes`, `archive_total_size_bytes` |
| RAG | `rag_indexed_at`, `rag_index_generation`, `rag_index_size_bytes`, `rag_indexed_record_count` |

The derivative writers use a 16-column subset of that surface — `visibility`, `source_mode`, `provider_metadata`,
`capture_metadata` and the governance trio among them (`scripts/mcp-server/umbrellas/dataset_admin.ts:1258-1262`).

Two facts an agent must not assume:

- **No parent/derived column exists on `registry.datasets`.** Derivation is recorded on the *file* row
  (`conduit.repo_files.parent_file_id / derivation_type / derivation_config / pipeline_hash`), not on the dataset
  (`docs/audit/portal-ontology/05-library-derivative-datasets.md:114-116`).
- **No workspace binding lives on the dataset row.** The Query-DB workspace resolves from `registry.dataset_loads`
  (latest load wins) then `registry.dataset_settings`; neither → typed `workspace_unresolved`
  (`lib/analytics/deepAnalysis.ts:65-79`).

Field-level metadata is a sibling table, `registry.dataset_fields`, populated by the analysis bridge
(`scripts/worker.ts:1030-1040`, `importAnalyticsProfile`) and read by `dataset_catalog` (`dataset_catalog.ts:63`).

---

## 2. Ingest paths

Four doors write datasets and files. All URL fetches are SSRF-guarded (`dataset_ingest.ts:821`).

| Door | Commands (measured case list) | What it accepts / does | Source |
|---|---|---|---|
| `dataset_ingest` | `ingest`, `cancel`, `url_ingest`, `batch_ingest`, `url_and_wait`, `upload_urls_get`, `load`, `unload`, `status_get` (9) | Inline content ≤10 MB (CSV/JSON, or PDF/DOCX/TXT/EML/HTML → `document_collection` dataset); URL download; async tabular batch (HEAD preflight sync, returns `batch_id`); `url_and_wait` tabular refresh orchestrator (job_id within 2 s); `load` into Query DB (`masking_mode='physical'` is async); `unload` is ADMIN + `confirm:true`. HTML pages are preflight-rejected by URL (LIB-BUG-210) — ingest inline. | `dataset_ingest.ts:819-840, 950-1021` |
| `file_repo` | `list`, `read_file`, `analysis_runs`, `derivatives_list`, `summary`, `versions_list`, `derivative_create`, `drift_events`, `drift_acknowledge`, `download_url`, `upload` (11) | Conduit file-repository access. Reads public; writes require API key. `upload` adds a file version; `derivative_create` stores caller-supplied bytes as a child of `parent_file_id` (§6). | `file_repo.ts:189-198, 322-480` |
| `crawler` | `project_*` (6), `source_*` (5), `portal_discover`, `dataset_refresh` (13) | Open-data portal crawls (socrata/arcgis/ckan/data_gov providers, `:150`), portal discovery over data.gov / global Socrata / known-portals registry (`:161`), and `dataset_refresh` → async `refresh_analyze` job (`:139`). `scope.sample_policy:'plain'` requires `acknowledge_plain_samples:true`, recorded as an audit row (`:150`). | `crawler.ts:123-162` |
| `rest` | `endpoint_create`, `endpoint_update`, `endpoint_delete`, `endpoint_list`, `execute`, `responses_diff` (6) | REST endpoint CRUD + execution; `auto_save` attaches the capture `file_repo` to the dataset (`:200`); optional cataloguing enqueues a discover scan on the account's `rest_api` catalog provider — refused `endpoint_unbound_not_catalogable` without `auth_connection_id` (`:89`). | `rest.ts:24-89, 207-540` |

Post-analysis loading is a single seam: `finalizeIngestAfterAnalysis` (`lib/ingest/postAnalysisLoad`, called at
`scripts/worker.ts:912-913` and `:940-941`) — auto-load and downstream steps run there, not inside each engine.

---

## 3. Analysis engines

Jobs are dispatched by `job_type` in `scripts/worker.ts`. The engine name stamped on the stored run is what
`dataset_analysis.runs_list` and `file_repo.analysis_runs` later report.

| `job_type` (dispatch line) | Engine name stamped | What runs |
|---|---|---|
| `refresh_analyze` (`worker.ts:317`) | inherits the analysis path | `refreshDataset` (`lib/conduit/crawlerDownload`) re-downloads by URL, then re-analyses and re-runs derivatives. |
| `dedup` (`worker.ts:422`) | `pg` (`:537`) | FEAT-013 async entity dedup in the Query DB against a composite `dedup_key` (FEAT-009 key groups, `:534`); writes a `_dedup_<ts>` derivative table and pre/post/removed counts. |
| `deep_analysis` (`worker.ts:615`) | — (writes `registry.rel_atlas`) | `runDeepAnalysis` → RelationshipAtlas (§3.1). `completeJob` with the consumed conduit `runId` (`:625`). |
| `rag_embedding` (`worker.ts:627`) | — (no engine) | **Declared but not implemented.** The branch throws `rag_embedding job type is not yet implemented` (`:628`; same in `worker-child.ts:193-196`), and the type is listed in `NOT_YET_IMPLEMENTED_JOB_TYPES` (`lib/conduit/analysisJobs.ts:46`), which is what removes it from the `dataset_analysis.submit` wire enum. RAG indexing runs on its own lane, not through this worker. |
| `full_analysis` / `reprocess` (`worker.ts:629`) | `duckdb_profiler` (`:728`) | Large CSV, **any-size Parquet** and routed JSON go to DuckDB: `computeRawProfile` + `formatProfile` (`lib/conduit/duckdbProfiler`, `:715-719`), stored via `storeAnalysisRun`, then bridged to `registry.dataset_fields`/`dataset_analytics_profiles` by `importAnalyticsProfile` with `engineName:"duckdb_profiler", engineVersion:"1.0"` (`:749`). LIB-BUG-269: small Parquet used to be dark because the in-memory engine cannot read `PAR1`. |
| same, document branch | `document_processor` (`:914`) / `document_disposition` (`:929`) | `analyzeFileBuffer` (`lib/conduit/analyzeFile`) text-extracts documents; a document with no extractable text stores a `document_disposition` run instead of failing (`:918-944`). |
| same, in-memory branch | `worker_processor` (`:1016`, `:1036`) | Default in-memory tabular analysis; bridged with `profileType:"conduit_analysis"` (`:1038`). |

Execution mode per account (`client` / `server` / `auto`) is read and set by `dataset_analysis.analysis_mode_get/set`
and honoured by the server-side paths only (`dataset_analysis.ts:1008`, `:1016`).

### 3.1 RelationshipAtlas — `deepAnalysis`

`lib/analytics/deepAnalysis.ts:1-24` states the contract, enforced in the module:

- Builds a RelationshipAtlas from the dataset's stored Conduit run plus governed correlation SQL; persists
  **atomically** — the only write is a single INSERT into `registry.rel_atlas` after the full build succeeds (`:9-10`).
- **Idempotent** per `(dataset_id, snapshot_ref, config-set hash)`; a matching live row is returned, never duplicated (`:7-8`, hash at `:83-90`).
- Edges capped top-k per node (`thresholds.atlas.max_edges_per_node`); suppressed relationships always persist — "they are the safety product" (`:11-12`).
- Guardrail outcomes surface as typed errors, never auto-forced (`:13`); every constant loads from the versioned
  analytics config store with hash-verified pins (`thresholds`, `relationship-policy`; `:98-104`).
- **Premise (META-CALIBRATION 5):** the DuckDB profiler stores an empty `pairwiseMatrix`, so correlations are computed
  via `corr()` through the masked-view query engine, bounded by `compute_row_limit`; r² = r·r (`:17-22`).
- Result shape: `{ atlasId, runId, reused, edgeCount, suppressedCount }` (`:55-61`).

### 3.2 The AnalysisContract sections

`config/analysis-contract-sections.config.ts:10-15` is the AnalysisContract v1.1 vocabulary (20 sections). Both engines
stamp the full list on `metadata.sectionsIncluded`; a partial producer (the catalog pushdown profile) stamps a declared
**subset** so a renderer shows which sections are absent instead of reading an empty section as "clean" (`:2-8`).
The config carries names only (17 lines; no per-section prose) — the descriptions below are the author's reading of
the producers in §3.

| Section | Carries |
|---|---|
| `executive-summary` | Headline row/column counts, quality verdict, top findings |
| `data-completeness` | Per-column null/missing percentages |
| `field-statistics`, `column-statistics-summary` | Per-column type, cardinality, min/max/quantiles (value-bearing members are withheld on public reads — §4.3) |
| `top-values` | Most frequent values per column (value-bearing; masked/withheld on egress) |
| `patterns-detected` | Classification hits from the pattern library (semantic types, PII/PHI) |
| `before-after`, `data-pipeline`, `original-sample`, `final-sample` | Transformation-pipeline effect and samples when a pipeline ran (JSON ingest path) |
| `dataset-profile`, `column-semantics` | Dataset-level shape; per-column semantic type used by `dataset_catalog.fields_search` |
| `relationship-graph` | Column relationships (deepAnalysis populates `registry.rel_atlas` separately) |
| `data-quality-metrics` | Quality score inputs (`registry.datasets.quality_score/quality_rating`) |
| `compliance-governance` | `compliance_level`, `has_pii`, `has_phi`, framework hits |
| `transformation-lineage`, `migration-playbook`, `business-rules` | Lineage narrative, import-SQL guidance, inferred rules |
| `metadata` | Engine name/version, `sectionsIncluded`, `vocabulary_version` (used by `submit_batch scope:'vocabulary_stale'`, `dataset_analysis.ts:1006`) |
| `source-fingerprint` | Source-system detection result (§9, `fingerprint`) |

---

## 4. Reading analysis and data

### 4.1 `dataset_analysis` (8 commands, `dataset_analysis.ts:998-1017`)

`get` (by `run_id > file_id > dataset_id` precedence, `:1028-1034`), `runs_list` (offset-paginated run history),
`create` (store an analysis result with `engine_name/engine_version`), `submit` (enqueue `full_analysis` /
`refresh_analyze` / `reprocess` / `dedup` / `deep_analysis` — the wire enum is `SUBMITTABLE_JOB_TYPES`, derived as
`JOB_TYPES` minus `NOT_YET_IMPLEMENTED_JOB_TYPES`, `:63-69`), `submit_batch` (ADMIN staged re-analysis; `dry_run` defaults true; scopes
`stale` / `all` / `vocabulary_stale`; DB-enforced idempotency against active jobs), `preflight` (HEAD+1 MB URL probe),
`analysis_mode_get/set`. Job lifecycle moved to the `job` umbrella (`:1017`). Annotated
`destructiveHint:true` because `submit(refresh_analyze)` replaces the stored file (`:1018-1024`).

Two state-source disagreements found while measuring this umbrella (META-CALIBRATION 5 — recorded, not resolved here):

- `create`'s own description says "store a **19-section** analysis result" (`:1004`), and `get`'s `summary_only`
  describes the full body as "**19-section**" (`:1035`) — while the AnalysisContract vocabulary both engines stamp
  carries **20** sections (§3.2). The prose predates `column-statistics-summary` / `source-fingerprint`; the config
  is the authority.
- `submit`'s description lists four job types (`:1005`); the wire enum admits five (`deep_analysis` included,
  `:63-69`). §10's `deep_analysis` row routes through `submit` and is correct.

### 4.2 `dataset_data` — the read-only query door (2 commands, `dataset_data.ts:307-336`)

- `query`: a single-statement `SELECT`/`WITH` against one dataset (`dataset_id`, cost-guardrailed, `force` override) or
  2+ datasets (`dataset_ids` + `aliases`, cross-schema). Engine routing is automatic — PostgreSQL for loaded datasets,
  DuckDB directly on the file for unloaded/large ones (`:311`). The read-only guard rejects forbidden keywords, second
  statements and file/network table functions (`:204`). Positional `$1` bind params work on every engine (`:327`).
- `sample`: headers + first N rows, masking applied, DuckDB fallback (`:312`).
- Canonical shape regardless of engine (LIB-BUG-439): `rows` keyed objects, authoritative `columns[]`, `types[]`,
  `totalRows`, `routing.engine` (`:313`).
- **Everything is served through masking views** (`:310`, `:328`). Loading is `dataset_ingest.load`; metadata search is `dataset_catalog` (`:314`).

### 4.3 `dataset_catalog` get / get_batch and the field-row projection

10 commands (`dataset_catalog.ts:113`): `search`, `get`, `get_batch`, `stats`, `tables`, `fields_search`,
`similar_list`, `change_history`, `catalog_facets`, `catalog_stats`. `get` returns fields/resources/lineage/quality plus a
`row_drift` signal (observed `row_count` vs provider `source_total_rows`, `:118`); `get_batch` does the same for 1–50 ids
(`:123`). Field rows come from `registry.dataset_fields` (`:63`).

**Field rows are value-free on this surface.** `sample_values` and value-bearing statistics (min/max/quantiles) are
withheld under the platform sample-value policy (default `omit`) because the catalog read is public-audience; each field
instead carries `sample_values_withheld {reason, policy}`; count statistics (missing %, uniqueCount, cardinality) are
returned in full (LIB-BUG-1395; `:185`, `:220`). The acknowledgement path that would release samples is unreachable
on this door — no door passes it (`:595`). `compact:true` trims each field to a short projection (`:263-264`, `:578-579`);
`include_fields:false` drops the list and keeps `fieldCount` (`:261-262`, `:575-577`). Column counts are reconciled
against the field list (LIB-BUG-416, `:258`).

---

## 5. Derivatives that create a NEW dataset — `dataset_admin.join_create` / `union_create`

`dataset_admin` has **13** commands (title `dataset_admin.ts:2444`; schema union `:2462-2602`; handler cases
`:2625-2705`) — `settings_get`, `settings_update`, `bulk_action`, `summary_refresh`, `masking_reload`, `masking_set`,
`changes_list`, `join_create`, `union_create`, `catalog_export`, `sync`, `refresh`, `columns_align`. The two
derivation verbs are `join_create` (`:2644`) and `union_create` (`:2645`).

| Property | `join_create` | `union_create` |
|---|---|---|
| Engine | in-memory DuckDB (`duckdb-async`, `Database.create(":memory:")`, `:796`, `:845-846`) | same |
| Reads | each source through its **served masked relation** — masked columns are materialised as masked tokens, never parent plaintext (`:2455`); a masked join key that is type-incompatible across sides refuses with a remedy (`:1419-1430`) | same masked-read contract (`:2456`, `:1573`) |
| Writes | new `registry.datasets` row (`:1257-1264`) + new `conduit.repo_files` row with `derivation_type='join'` (`:1300-1305`) | new dataset (`:2110-2131`) + `repo_files` row (`:2147-2152`) |
| Governance | born with strictest-of-parents `compliance_level/has_pii/has_phi` + framework union (`:2455`) | same (`:1554`) |
| Cardinality guard | estimates output rows; refuses `join_cardinality_exceeded` unless `confirm_large_output:true` (`:2517`) | — |
| Analysis re-enqueued | `enqueueAnalysisJob` `full_analysis` unless `auto_analyze:false` (`:1356-1360`) | `full_analysis` priority 5 + optional async `dedup` job (`:2186-2199`) |
| **Parent link** | **JSONB only**: `derivation_config.join_sources[{dataset_id,file_id,role}]` (`:1308-1312`); `parent_file_id` is NULL | `derivation_config.union_sources[...]` (`:2156`); `parent_file_id` NULL |

Consequence measured in `docs/audit/portal-ontology/05-library-derivative-datasets.md:125-126`: the 17 live new-dataset
derivatives (6 join, 11 union) have `parent_file_id NULL`; `registry.asset_edges` holds 0 dataset→dataset rows and
`derived_from` is declared with no writer (`:118-119`) — so the lineage canvas renders a join/union output with no
upstream (`:128-132`). The OpenLineage store (M-1/M-2) is the planned fix, not a new materialiser (`:133-135`).

---

## 6. Derivatives inside an existing repo — `file_repo.derivative_create`

`file_repo.derivative_create` (`file_repo.ts:403-442`) stores **caller-supplied** base64 bytes as a new
`conduit.repo_files` version under `parent_file_id` with `derivation_type/derivation_config`, optional analysis attach
(`:198`, `:251`). The platform computes nothing here — it records what the caller computed (`05-...md:148`). It is
dual-target gated (repo mutate + parent-file read, LIB-BUG-407; `:316-317`); a concurrent version allocation returns a
retryable conflict (`:509`). No new dataset is created.

---

## 7. The declarative transform model — `dataset_transform`

### 7.1 Vocabulary (`config/transform-model.config.ts`)

`TRANSFORM_TYPES` (`:55-77`) — exactly 21, asserted at `:797`:
`remove_column, rename_column, add_column, reorder_columns, filter, mask_pii, standardize_date, type_cast, deduplicate,
aggregate, join, validate, custom_sql, custom_python, fill_null, trim, uppercase, lowercase, regex_replace,
split_column, merge_columns`.

| Config | Value (measured) |
|---|---|
| `TRANSFORM_STATUSES` (`:92`) | `supported`, `narrowed`, `extension`, `unsupported` |
| `TRANSFORM_STATUS_CENSUS` (`:701-705`) | supported 9 · narrowed 3 · extension 5 · unsupported 4 |
| `TRANSFORM_KINDS` (`:102`) | `column`, `header`, `row_set`, `derived` |
| `CODEGEN_CENSUS.pandas` (`:714-715`) | emit 17 · recipe 2 (`aggregate`, `join`) · refuse 2 (`custom_sql`, `custom_python`) |
| `VALIDATION_RULE_ACTIONS` / `_SEVERITIES` (`:693-694`) | `log` \| `filter` \| `fail` — and `warning` \| `error` \| `fatal` |
| `EVALUATION_FAILURE_POLICY.mode` (`:751-752`) | `count_and_report` (alt. `fail_closed`) |
| Guards | `REGEX_GUARD` (`:254`), `COLUMN_IDENTIFIER_RE` + reserved list (`:233-241`), `COLUMN_NAME_WIRE_CLAMP=128` (`:231`), `PREVIEW_CONTRACT` (`:290`) |

Each type has a `TRANSFORM_MODEL[type]` spec (`:358-690`) and the module self-checks duplicates, model coverage and
the census on load (`:793-810`).

### 7.2 Commands (`dataset_transform.ts:113-244`)

`json_to_csv`, `json_process`, `transform_preview`, `transform_recipe_list`, `transform_recipe_get`, `transform_codegen`.

- **`transform_preview`** (`:198`) — evaluates the pipeline in memory and returns what *would* change: column list,
  row/column counts, per-column before/after **shape tokens**, change summary, flatten-spec and frame digests. Returns
  **no cell values by construction** and refuses if its own output scan finds one (`:42`). Unsupported step types
  (`aggregate`, `join`, `custom_sql`, `custom_python`) are refused with the reason, never dropped (`:46`).
- **`transform_codegen`** (`:244`) — generates a complete pandas source file (helper prelude, named-parameter block,
  pipeline, input-column assertion) plus attribution rows. Every caller literal is bound to a named parameter, never
  interpolated. Expression constructs and bare regex literals with no faithful `re` form refuse the whole request rather
  than emit code that computes something else (`:43`). `attach` writes the file as a derivative
  (`text/x-python`, `derivation_type transform_codegen`) into a repo the caller owns; `parent_file_id` must belong to
  `repo_id` (`:81-82`); ownership is checked before a byte is generated (`:104`).

### 7.3 P-3 — the no-execution invariant

"GENERATE-ONLY: the platform writes the file and NEVER runs it; there is no Python runtime, interpreter or sandbox on
this surface" (`dataset_transform.ts:43`, `:245`). The invariant lives in `config/codegen-no-execution.config.ts` (`05-...md:77`).
Note from the same audit (`:80`): no `audit:codegen-no-execution` gate exists in `package.json` — prior docs naming it
as a gate were wrong; `test:transform-engine` and `test:transform-bundle-smoke` are the chained tests (`package.json:98`).

### 7.4 Honest statement — the model does not materialise a dataset

Per `docs/audit/portal-ontology/05-library-derivative-datasets.md:26,58-59,137-146`: there is **no `transform_apply`
/ `materialize` verb and no job kind runs `applyTransformations`**. `transform_preview` is a simulator;
`transform_codegen` produces a real artefact that is never executed; the only path that applies the 21-type pipeline to
rows is `POST /api/conduit/json/ingest` (REST, JSON input only), which writes a derivative **file** in the parent repo —
not a dataset — and has 0 recorded uses (`:143`, `:66`). The only verbs that persist a *new dataset* from existing ones are
`join_create` and `union_create` (§5).

---

## 8. Snapshots, lifecycle, publish / sync / archive / export

**`dataset_snapshot`** (7 commands, `dataset_snapshot.ts:33-43`): point-in-time snapshots of the loaded source table.
`create` (server prefixes `_snapshot_`, bare label and handle both resolve), `list` (+ retention summary), `get`,
`compare` (row-level diff, natural key auto-resolved from analysis else synthetic `_row_id` with a load-order caveat, `:54`),
`schema_diff` (added/removed/renamed columns + type changes), `delete` and `restore` (zero-downtime swap, irreversible,
`confirm:true`). **Not a changeset reader:** `dataset_admin.changes_list` (`dataset_admin.ts:2640` →
`handleAdminChangesList`, `:474-494`) runs `queryCdcLog` over the *loaded* schema's CDC log and refuses
`dataset_not_loaded` otherwise — it does **not** read the `changeset_computed` derivative files auto-diff writes
(`05-...md:90`, `:147`); nothing on the MCP surface reads those today.

**`dataset_lifecycle`** (6 commands, `dataset_lifecycle.ts:360-370`): `delete` (soft default; hard = `DROP SCHEMA CASCADE`,
`confirm:true`, `force` on certified/reference, and `mdm_disposition retain|purge` REQUIRED — LIB-BUG-569, `:379`),
`undelete` (refuses when content was erased, `:533-541`, or the dataset is hollow, `:587-598`), `archive_create` (S3/B2/R2/
Wasabi/Spaces/MinIO, optional `drop_after`), `archive_list`, `archive_delete`, `restore`. All commands take canonical
`dataset_id`; bare `id` strict-rejects (`:364`). Stages: there is no `dataset_admin.set_lifecycle_stage` command — the stage is set either by
`dataset_admin.settings_update {lifecycle_stage}` (`dataset_admin.ts:2475`) or by
`dataset_admin.bulk_action {action:'set_lifecycle_stage'}` (`BULK_ACTIONS`, `lib/lifecycle/bulkActions.ts:15-23`),
both over the same closed enum `certified | reference | published | development | test | archived | uncategorized`
(`config/lifecycle.config.ts:17-25`).

**Publish / sync / export** (`dataset_admin.ts:2457-2458`):
- `sync` — publish a governed copy to a remote S3-compatible connection: metadata sidecars
  (dataset / schema / lineage / quality / governance / import SQL / README / manifest) plus by default the data file;
  `apply_masking` / `row_limit` / `columns` apply to the uploaded file and `governance.json`'s `data_file` claim states
  exactly what was applied; a destination that cannot honour requested encryption refuses
  `archive_encryption_unavailable` rather than upload cleartext (`:611`, `:2458`).
- `catalog_export` — Unity Catalog format by design, no format selector (`:2457`, `:2646`).
- `05-...md:150` records these as real writers with **no catalogued run** (W1) — sync/archive/export leave no lineage run today.

---

## 9. Generalization, fingerprint, evidence

**`generalization`** (7 commands, `generalization.ts:136-147`) — privacy-preserving column transforms. `view` mode is
reversible masking applied at query / prompt-egress time and works; **`physical` mode returns `not_implemented`**
(FEAT-033, `:138`, `:141`). Strategies and constrained options: `dob_generalize` (year|decade|age_range),
`zip_generalize` (3|4 digits), `age_band`, `phone_partial`, `ssn_partial`, `email_domain` (`:147`). `preview` needs a
loaded dataset; `stats` returns utility score + compliance coverage.

**`fingerprint`** (6 commands, `fingerprint.ts:71-84`) — identify the source system (Salesforce, SAP, Workday, …) from
column names, sample values and metadata; 95 built-in + custom definitions (`:77`). `detect` scoring is IDF-weighted with
calibrated `confidence`, raw `score` as the ranking key, `matchedSignals` and a `basis` string (`:79`); `validate`
dry-runs ReDoS/length/count checks; `create`/`delete` are Developer tier+, creator-or-admin scoped. Feeds the
`source-fingerprint` contract section (§3.2) and `registry.datasets.source_system`.

**`evidence`** (4 commands, `evidence.ts:20-26`) — the "prove" leg of find→prove→animate→watch. Rows are written at
`story.scan`, pin their snapshot in the retention index and hash the payload over RFC 8785 canonical JSON. `show`
(`variant:'public'` re-serves through the egress gate; PHI datasets refuse public without a §164.514(b)(1)
determination), `verify` (re-hash vs `content_hash`), `list` (per claim), `determine` (admin; append-only PHI-publication
determination; revocation is a new row, `:26`, `:27-32`).

---

## 10. Quick routing table for agents

| Need | Call |
|---|---|
| Add data | `dataset_ingest.ingest` / `url_ingest` / `batch_ingest`; documents inline via `ingest` |
| Make rows queryable | `dataset_ingest.load` → `dataset_data.query` / `sample` (masked) |
| Metadata, fields (value-free) | `dataset_catalog.get` / `get_batch` / `fields_search` |
| Run or read analysis | `dataset_analysis.submit` (poll `job.get`) / `get` / `runs_list` |
| Relationships | `dataset_analysis.submit {job_type:'deep_analysis'}` → `registry.rel_atlas` |
| New dataset from two or more | `dataset_admin.join_create` / `union_create` (DuckDB, masked-read, analysis re-enqueued) |
| Preview or generate a pipeline | `dataset_transform.transform_preview` / `transform_codegen` — nothing executes (P-3), nothing materialises |
| Store a computed file | `file_repo.derivative_create` (caller-computed bytes) |
| Point-in-time / diff | `dataset_snapshot.create` / `compare` / `schema_diff` |
| Publish a governed copy | `dataset_admin.sync` (sidecars + masked data file) |
| Cold storage / delete | `dataset_lifecycle.archive_create` / `delete` (hard requires `mdm_disposition`) |