Datasets, Analysis and Transforms

On this page

• 1. The dataset model — registry.datasets
• 2. Ingest paths
• 3. Analysis engines
• 4. Reading analysis and data
• 5. Derivatives that create a NEW dataset — dataset_admin.join_create / union_create
• 6. Derivatives inside an existing repo — file_repo.derivative_create
• 7. The declarative transform model — dataset_transform
• 8. Snapshots, lifecycle, publish / sync / archive / export
• 9. Generalization, fingerprint, evidence
• 10. Quick routing table for agents

1. The dataset model — registry.datasets

The dataset is the unit everything else attaches to. Its row was created in migrations/registry/0001_init.sql:102-142 and extended by later migrations. The base columns:

Later migrations add 53 more columns by ALTER TABLE registry.datasets ADD COLUMN (measured across migrations/registry/*.sql; none is ever dropped):

The derivative writers use a 16-column subset of that surface — visibility, source_mode, provider_metadata, capture_metadata and the governance trio among them (scripts/mcp-server/umbrellas/dataset_admin.ts:1258-1262).

Two facts an agent must not assume:

• No parent/derived column exists on registry.datasets. Derivation is recorded on the file row

(conduit.repo_files.parent_file_id / derivation_type / derivation_config / pipeline_hash), not on the dataset (docs/audit/portal-ontology/05-library-derivative-datasets.md:114-116).

• No workspace binding lives on the dataset row. The Query-DB workspace resolves from registry.dataset_loads

(latest load wins) then registry.dataset_settings; neither → typed workspace_unresolved (lib/analytics/deepAnalysis.ts:65-79).

Field-level metadata is a sibling table, registry.dataset_fields, populated by the analysis bridge (scripts/worker.ts:1030-1040, importAnalyticsProfile) and read by dataset_catalog (dataset_catalog.ts:63).

2. Ingest paths

Four doors write datasets and files. All URL fetches are SSRF-guarded (dataset_ingest.ts:821).

Post-analysis loading is a single seam: finalizeIngestAfterAnalysis (lib/ingest/postAnalysisLoad, called at scripts/worker.ts:912-913 and :940-941) — auto-load and downstream steps run there, not inside each engine.

3. Analysis engines

Jobs are dispatched by job_type in scripts/worker.ts. The engine name stamped on the stored run is what dataset_analysis.runs_list and file_repo.analysis_runs later report.

Execution mode per account (client / server / auto) is read and set by dataset_analysis.analysis_mode_get/set and honoured by the server-side paths only (dataset_analysis.ts:1008, :1016).

3.1 RelationshipAtlas — deepAnalysis

lib/analytics/deepAnalysis.ts:1-24 states the contract, enforced in the module:

• Builds a RelationshipAtlas from the dataset's stored Conduit run plus governed correlation SQL; persists

atomically — the only write is a single INSERT into registry.rel_atlas after the full build succeeds (:9-10).

• Idempotent per (dataset_id, snapshot_ref, config-set hash); a matching live row is returned, never duplicated (:7-8, hash at :83-90).
• Edges capped top-k per node (thresholds.atlas.max_edges_per_node); suppressed relationships always persist — "they are the safety product" (:11-12).
• Guardrail outcomes surface as typed errors, never auto-forced (:13); every constant loads from the versioned

analytics config store with hash-verified pins (thresholds, relationship-policy; :98-104).

• Premise (META-CALIBRATION 5): the DuckDB profiler stores an empty pairwiseMatrix, so correlations are computed

via corr() through the masked-view query engine, bounded by compute_row_limit; r² = r·r (:17-22).

• Result shape: { atlasId, runId, reused, edgeCount, suppressedCount } (:55-61).

3.2 The AnalysisContract sections

config/analysis-contract-sections.config.ts:10-15 is the AnalysisContract v1.1 vocabulary (20 sections). Both engines stamp the full list on metadata.sectionsIncluded; a partial producer (the catalog pushdown profile) stamps a declared subset so a renderer shows which sections are absent instead of reading an empty section as "clean" (:2-8). The config carries names only (17 lines; no per-section prose) — the descriptions below are the author's reading of the producers in §3.

4. Reading analysis and data

4.1 dataset_analysis (8 commands, dataset_analysis.ts:998-1017)

get (by run_id > file_id > dataset_id precedence, :1028-1034), runs_list (offset-paginated run history), create (store an analysis result with engine_name/engine_version), submit (enqueue full_analysis / refresh_analyze / reprocess / dedup / deep_analysis — the wire enum is SUBMITTABLE_JOB_TYPES, derived as JOB_TYPES minus NOT_YET_IMPLEMENTED_JOB_TYPES, :63-69), submit_batch (ADMIN staged re-analysis; dry_run defaults true; scopes stale / all / vocabulary_stale; DB-enforced idempotency against active jobs), preflight (HEAD+1 MB URL probe), analysis_mode_get/set. Job lifecycle moved to the job umbrella (:1017). Annotated destructiveHint:true because submit(refresh_analyze) replaces the stored file (:1018-1024).

Two state-source disagreements found while measuring this umbrella (META-CALIBRATION 5 — recorded, not resolved here):

• create's own description says "store a 19-section analysis result" (:1004), and get's summary_only

describes the full body as "19-section" (:1035) — while the AnalysisContract vocabulary both engines stamp carries 20 sections (§3.2). The prose predates column-statistics-summary / source-fingerprint; the config is the authority.

• submit's description lists four job types (:1005); the wire enum admits five (deep_analysis included,

:63-69). §10's deep_analysis row routes through submit and is correct.

4.2 dataset_data — the read-only query door (2 commands, dataset_data.ts:307-336)

• query: a single-statement SELECT/WITH against one dataset (dataset_id, cost-guardrailed, force override) or

2+ datasets (dataset_ids + aliases, cross-schema). Engine routing is automatic — PostgreSQL for loaded datasets, DuckDB directly on the file for unloaded/large ones (:311). The read-only guard rejects forbidden keywords, second statements and file/network table functions (:204). Positional $1 bind params work on every engine (:327).

• sample: headers + first N rows, masking applied, DuckDB fallback (:312).
• Canonical shape regardless of engine (LIB-BUG-439): rows keyed objects, authoritative columns[], types[],

totalRows, routing.engine (:313).

• Everything is served through masking views (:310, :328). Loading is dataset_ingest.load; metadata search is dataset_catalog (:314).

4.3 dataset_catalog get / get_batch and the field-row projection

10 commands (dataset_catalog.ts:113): search, get, get_batch, stats, tables, fields_search, similar_list, change_history, catalog_facets, catalog_stats. get returns fields/resources/lineage/quality plus a row_drift signal (observed row_count vs provider source_total_rows, :118); get_batch does the same for 1–50 ids (:123). Field rows come from registry.dataset_fields (:63).

Field rows are value-free on this surface. sample_values and value-bearing statistics (min/max/quantiles) are withheld under the platform sample-value policy (default omit) because the catalog read is public-audience; each field instead carries sample_values_withheld {reason, policy}; count statistics (missing %, uniqueCount, cardinality) are returned in full (LIB-BUG-1395; :185, :220). The acknowledgement path that would release samples is unreachable on this door — no door passes it (:595). compact:true trims each field to a short projection (:263-264, :578-579); include_fields:false drops the list and keeps fieldCount (:261-262, :575-577). Column counts are reconciled against the field list (LIB-BUG-416, :258).

5. Derivatives that create a NEW dataset — dataset_admin.join_create / union_create

dataset_admin has 13 commands (title dataset_admin.ts:2444; schema union :2462-2602; handler cases :2625-2705) — settings_get, settings_update, bulk_action, summary_refresh, masking_reload, masking_set, changes_list, join_create, union_create, catalog_export, sync, refresh, columns_align. The two derivation verbs are join_create (:2644) and union_create (:2645).

Consequence measured in docs/audit/portal-ontology/05-library-derivative-datasets.md:125-126: the 17 live new-dataset derivatives (6 join, 11 union) have parent_file_id NULL; registry.asset_edges holds 0 dataset→dataset rows and derived_from is declared with no writer (:118-119) — so the lineage canvas renders a join/union output with no upstream (:128-132). The OpenLineage store (M-1/M-2) is the planned fix, not a new materialiser (:133-135).

6. Derivatives inside an existing repo — file_repo.derivative_create

file_repo.derivative_create (file_repo.ts:403-442) stores caller-supplied base64 bytes as a new conduit.repo_files version under parent_file_id with derivation_type/derivation_config, optional analysis attach (:198, :251). The platform computes nothing here — it records what the caller computed (05-...md:148). It is dual-target gated (repo mutate + parent-file read, LIB-BUG-407; :316-317); a concurrent version allocation returns a retryable conflict (:509). No new dataset is created.

7. The declarative transform model — dataset_transform

7.1 Vocabulary (config/transform-model.config.ts)

TRANSFORM_TYPES (:55-77) — exactly 21, asserted at :797: `remove_column, rename_column, add_column, reorder_columns, filter, mask_pii, standardize_date, type_cast, deduplicate, aggregate, join, validate, custom_sql, custom_python, fill_null, trim, uppercase, lowercase, regex_replace, split_column, merge_columns`.

Each type has a TRANSFORM_MODEL[type] spec (:358-690) and the module self-checks duplicates, model coverage and the census on load (:793-810).

7.2 Commands (dataset_transform.ts:113-244)

json_to_csv, json_process, transform_preview, transform_recipe_list, transform_recipe_get, transform_codegen.

• transform_preview (:198) — evaluates the pipeline in memory and returns what would change: column list,

row/column counts, per-column before/after shape tokens, change summary, flatten-spec and frame digests. Returns no cell values by construction and refuses if its own output scan finds one (:42). Unsupported step types (aggregate, join, custom_sql, custom_python) are refused with the reason, never dropped (:46).

• transform_codegen (:244) — generates a complete pandas source file (helper prelude, named-parameter block,

pipeline, input-column assertion) plus attribution rows. Every caller literal is bound to a named parameter, never interpolated. Expression constructs and bare regex literals with no faithful re form refuse the whole request rather than emit code that computes something else (:43). attach writes the file as a derivative (text/x-python, derivation_type transform_codegen) into a repo the caller owns; parent_file_id must belong to repo_id (:81-82); ownership is checked before a byte is generated (:104).

7.3 P-3 — the no-execution invariant

"GENERATE-ONLY: the platform writes the file and NEVER runs it; there is no Python runtime, interpreter or sandbox on this surface" (dataset_transform.ts:43, :245). The invariant lives in config/codegen-no-execution.config.ts (05-...md:77). Note from the same audit (:80): no audit:codegen-no-execution gate exists in package.json — prior docs naming it as a gate were wrong; test:transform-engine and test:transform-bundle-smoke are the chained tests (package.json:98).

7.4 Honest statement — the model does not materialise a dataset

Per docs/audit/portal-ontology/05-library-derivative-datasets.md:26,58-59,137-146: there is **no transform_apply / materialize verb and no job kind runs applyTransformations**. transform_preview is a simulator; transform_codegen produces a real artefact that is never executed; the only path that applies the 21-type pipeline to rows is POST /api/conduit/json/ingest (REST, JSON input only), which writes a derivative file in the parent repo — not a dataset — and has 0 recorded uses (:143, :66). The only verbs that persist a new dataset from existing ones are join_create and union_create (§5).

8. Snapshots, lifecycle, publish / sync / archive / export

dataset_snapshot (7 commands, dataset_snapshot.ts:33-43): point-in-time snapshots of the loaded source table. create (server prefixes _snapshot_, bare label and handle both resolve), list (+ retention summary), get, compare (row-level diff, natural key auto-resolved from analysis else synthetic _row_id with a load-order caveat, :54), schema_diff (added/removed/renamed columns + type changes), delete and restore (zero-downtime swap, irreversible, confirm:true). Not a changeset reader: dataset_admin.changes_list (dataset_admin.ts:2640 → handleAdminChangesList, :474-494) runs queryCdcLog over the loaded schema's CDC log and refuses dataset_not_loaded otherwise — it does not read the changeset_computed derivative files auto-diff writes (05-...md:90, :147); nothing on the MCP surface reads those today.

dataset_lifecycle (6 commands, dataset_lifecycle.ts:360-370): delete (soft default; hard = DROP SCHEMA CASCADE, confirm:true, force on certified/reference, and mdm_disposition retain|purge REQUIRED — LIB-BUG-569, :379), undelete (refuses when content was erased, :533-541, or the dataset is hollow, :587-598), archive_create (S3/B2/R2/ Wasabi/Spaces/MinIO, optional drop_after), archive_list, archive_delete, restore. All commands take canonical dataset_id; bare id strict-rejects (:364). Stages: there is no dataset_admin.set_lifecycle_stage command — the stage is set either by dataset_admin.settings_update {lifecycle_stage} (dataset_admin.ts:2475) or by dataset_admin.bulk_action {action:'set_lifecycle_stage'} (BULK_ACTIONS, lib/lifecycle/bulkActions.ts:15-23), both over the same closed enum certified | reference | published | development | test | archived | uncategorized (config/lifecycle.config.ts:17-25).

Publish / sync / export (dataset_admin.ts:2457-2458):

• sync — publish a governed copy to a remote S3-compatible connection: metadata sidecars

(dataset / schema / lineage / quality / governance / import SQL / README / manifest) plus by default the data file; apply_masking / row_limit / columns apply to the uploaded file and governance.json's data_file claim states exactly what was applied; a destination that cannot honour requested encryption refuses archive_encryption_unavailable rather than upload cleartext (:611, :2458).

• catalog_export — Unity Catalog format by design, no format selector (:2457, :2646).
• 05-...md:150 records these as real writers with no catalogued run (W1) — sync/archive/export leave no lineage run today.

9. Generalization, fingerprint, evidence

generalization (7 commands, generalization.ts:136-147) — privacy-preserving column transforms. view mode is reversible masking applied at query / prompt-egress time and works; physical mode returns not_implemented (FEAT-033, :138, :141). Strategies and constrained options: dob_generalize (year|decade|age_range), zip_generalize (3|4 digits), age_band, phone_partial, ssn_partial, email_domain (:147). preview needs a loaded dataset; stats returns utility score + compliance coverage.

fingerprint (6 commands, fingerprint.ts:71-84) — identify the source system (Salesforce, SAP, Workday, …) from column names, sample values and metadata; 95 built-in + custom definitions (:77). detect scoring is IDF-weighted with calibrated confidence, raw score as the ranking key, matchedSignals and a basis string (:79); validate dry-runs ReDoS/length/count checks; create/delete are Developer tier+, creator-or-admin scoped. Feeds the source-fingerprint contract section (§3.2) and registry.datasets.source_system.

evidence (4 commands, evidence.ts:20-26) — the "prove" leg of find→prove→animate→watch. Rows are written at story.scan, pin their snapshot in the retention index and hash the payload over RFC 8785 canonical JSON. show (variant:'public' re-serves through the egress gate; PHI datasets refuse public without a §164.514(b)(1) determination), verify (re-hash vs content_hash), list (per claim), determine (admin; append-only PHI-publication determination; revocation is a new row, :26, :27-32).

10. Quick routing table for agents

Measured against the release named in the text. Raw Markdown: /ontology/documentation/datasets-analysis-transforms.md. This page is also served as raw Markdown at the same URL with a .md suffix, for agents and search tools that prefer plain text. Every capability statement cites the source file and line it was measured from.