Data Catalog

On this page

• 1. What the Data Catalog is
• 2. Command reference
• 3. Provider kinds — what each implements today
• 4. Sample-value policy ladder
• 5. Scan purposes and the blast-radius sweep guard
• 6. Quality scoring at read time
• 7. The Data Catalog app
• 8. REST bridge
• 9. Common errors and their fix hints
• 10. Trust notes

1. What the Data Catalog is

The Data Catalog is the platform's system-object estate map: registered providers (a database, warehouse, SaaS API, storage bucket or open-data portal), the objects (tables, views, schemas, endpoints…) a scan discovered under them, the columns of those objects with their type/class/constraints, optional profiles, business-term glossary, and bindings that feed catalogued objects into MDM mastering.

It is exposed on three parity surfaces built from ONE contract:

Tier floor: the umbrella is a developer-tier addition (config/mcp-tiers.config.ts:186-187, in DEVELOPER_TIER_ADDITIONS declared at line 174). Per-command floors above that are in config/command-auth.config.ts:551-564; ownership gates (owner-or-admin, per-account rows) are in config/authorization-policy.config.ts:556 (UMBRELLA_COMMAND_GATES.data_catalog) and run BEFORE the switch (umbrella header line 9).

2. Command reference

Verbs are derived, never hand-listed (DATA_CATALOG_VERBS, wire.ts:328). 29 commands (counted from the discriminated union; DATA_CATALOG_VERB_METHODS carries the same 29 keys — 15 GET, 14 POST). "Floor" = developer tier unless admin is stated (command-auth.config.ts:551-564). REST method per verb from DATA_CATALOG_VERB_METHODS (wire.ts:411-419).

2.1 Providers

2.2 Crawlers and scans

2.3 Reading the estate

2.4 Tags

Tag grammar (ONE sentence, ASSET_TAG_RULE_SENTENCE at config:807-809, rendered from ASSET_TAGS, config:781-792): lowercase, 1–64 chars, starts with a letter or digit, may contain ., :, -, _ inside (e.g. pii:reviewed, team.finance). The DB CHECK asset_tags_tag_check (migration 0235) retypes the same regex; audit:catalog-config-agreement binds the two (config:795-797).

2.5 Glossary

2.6 Bindings and remaster (MDM seam)

These reach the same writers the mdm umbrella admin-floors, so they are admin-floored too ("a second door must not be a lower floor than the first", command-auth.config.ts:554-558).

2.7 REST-endpoint capture and enrichers

2.8 Typical sequence

provider_create {kind, name, connection_id}      → provider_id
provider_test   {provider_id}                    → health persisted (dial/credential classified)
crawler_create  {provider_id, purpose:"discover", name, scope?} → crawler_project_id
scan_run        {crawler_project_id}             → job enqueued (job.* to follow it)
assets_list     {provider_id}                    → asset_id …
asset_get / columns_list {asset_id}
crawler_create  {purpose:"profile", scope:{profiling_mode:"M3"}} → scan_run → profile_get {asset_id}
bind_inbound    {asset_id, entity_type, field_map}   (admin; needs an M3 extract dataset — see errors)

3. Provider kinds — what each implements today

Source: PROVIDER_KIND_DECLARATIONS → PROVIDER_KINDS (config/data-catalog.config.ts:227-477, derivation at 483); 26 declared kinds. "Identity" below is the row's auth_source field (config:119). path follows STANDARD-010 §1: A = native dial with a declared driver module, B = REST + vault credential, ingest = reached through a pre-existing ingest/crawler lane (not the catalog's own dial). implemented flags are {discover, profile, extract, write, stream}; write/stream are false for every kind in this release.

A declared-but-unbuilt kind is admitted by the enum but provider_create/scan_run refuse with provider_kind_not_implemented / provider_not_scannable (§9). Harvest lanes on disk: lib/data_catalog/harvest/{postgres,snowflake,databricks,bigquery,sqlserver,mysql,rest_api,s3,s3Lake,saas,salesforce,dynamics365}.ts; profile lanes: lib/data_catalog/profile/{postgres,snowflake,databricks,bigquery,sqlserver,mysql}.ts; extract lanes: lib/data_catalog/extract/{postgres,rest_api}.ts plus extractDialects.ts (other SQL engines share the dialect table — inferred from filenames, not read).

4. Sample-value policy ladder

SAMPLE_VALUE_POLICIES = ["plain","masked","omit"] (config:1110). Default omit — no source values stored. Standards cited on the wire: GDPR Art. 5(1)(c), Art. 25, HIPAA 45 CFR 164.502(b) (config:1135). Detection (PII pattern label, lib/data_catalog/piiLabel.ts) ALWAYS runs; only PERSISTING values is gated (config:1107-1108).

Resolution ladder, most specific wins (SAMPLE_VALUE_POLICY.ladder, config:1128): column → scope (crawler) → provider → platform (admin.settings_set key data_catalog.sample_value_policy) → config default. Resolver: resolveSamplePolicy (lib/data_catalog/policyLadder.ts:72) returns {policy, source} so every read can say which rung decided.

Acknowledgement: plain is in requiresAcknowledgement (config:1133). Setting it at provider_create/provider_update or in a crawler scope REQUIRES acknowledge_plain_samples: true; the acknowledgement is written to the job log as an audit row. Reading stored plain values back (columns_list, profile_get) again requires acknowledge_plain_samples: true on the read. The UI's Acknowledgements section (§7) lists providers holding values in the clear and the recorded decision.

5. Scan purposes and the blast-radius sweep guard

Purposes (config:910): discover (harvest objects/columns), profile (column statistics into an AnalysisContract, modes M0–M3), extract (read a relation's ROWS into a dataset for mastering), rescan. Handlers: lib/data_catalog/scan/{run,handlers}.ts; applier lib/data_catalog/harvest/apply.ts.

Sweep guard (SWEEP_GUARD, config:3066-3069): after a harvest, objects no longer seen would be soft-deleted ("swept"). The applier REFUSES the sweep (sweep_refused, apply.ts:263-271) when (a) the harvest saw ZERO objects while the estate still held live ones (zero_objects), or (b) the estate before had ≥ 5 live objects (minEstateForFractionGuard) and the delete would exceed 50 % of it (maxFractionDeleted). Reasons vocabulary SWEEP_GUARD_REASONS = ["zero_objects","magnitude"] (config:3063). Contract violations (harvester spelled scanned_schemas in an undeclared vocabulary) are a SEPARATE, non-ratifiable vocabulary (config:3072-3080).

Acknowledgement (SWEEP_ACKNOWLEDGEMENT, config:3189+): scan_run {acknowledge_sweep:true, reason} — reason 12–500 chars; persisted under catalog.scans.scope.acknowledge_sweep; consumed once; expires after the declared ISO 8601 TTL PT1H (ttl, config:3203); bound to the crawler project AND to the principal the job is attributed to; not carried through job.retry (discloses acknowledgement_not_carried). An expired/foreign/malformed carried acknowledgement is refused with sweep_acknowledgement_rejected — never silently ignored (wire.ts scan_run describe).

6. Quality scoring at read time

quality_basis is ALWAYS derived_at_read (QUALITY_BASIS, config/asset-quality.config.ts:253; wire wire.ts:373): the headline overall (0–100 or null) is projected from stored per-axis measurements and the profile's sampling denominator by projectAssetQuality (lib/data_catalog/profile/qualityProjection.ts:140) — a stored headline is never served, and a 0-row profile reads as unmeasurable estate-wide with no backfill. assets_list returns the compact HEADLINE projection only — no prior contract, so no trend on a list page (assets.ts:104); asset_get carries the per-axis table and compares against the prior profile for trend (assets.ts:163-164). When no honest headline exists, quality_unavailable_reason names why (QUALITY_UNAVAILABLE_REASONS, asset-quality.config.ts:183, e.g. extract_quality_in_dataset_store, extract_backing_dataset_unrecorded — dispatch.ts:744). Privacy runs first: withholdUnmeasuredContractScores (dispatch.ts:777-816) strips producer-side scores the kernel did not measure.

7. The Data Catalog app

Route /app/data-catalog (app/app/data-catalog/page.tsx → DataCatalogDashboard.tsx). Sections are data in config/data-catalog-nav.config.ts:181-212 (order IS the nav) and bound to components in app/app/data-catalog/sections/index.ts:35-47; checkSectionRegistry fails at module load if any declared section lacks a component. Each row declares its feed verb, non-admin writes, adminWrites (⊆ the admin-floored commands) and needsProvider; test:data-catalog-shell L6 asserts those against the live config.

Lineage page — app/data-catalog/lineage/page.tsx (segment LINEAGE_ROUTE_SEGMENT = "lineage", label "Lineage", config/lineage-view.config.ts:348-351; nav row at config/app-nav.config.ts:464). It is anchored on one asset (?start=<type>:<id>, parsed by lib/server/lineage/contract.ts) and draws the derivation tree + stored relationships via components/lineage/LineageCanvas. Its honest-empty state is deliberate data, not a defect (LINEAGE_EMPTY_COPY, lineage-view.config.ts:368-377): noAnchor — "Pick something to trace"; noLayers — "No derived files and no snapshots": a typical sub-1GB CSV genuinely has one original file and no derivation layer; layers appear when something derives from it — dataset_snapshot.create writes a snapshot, and the declarative JSON ingest writes a transform step per pipeline operation. Rendered through components/flow/ui HonestEmpty. The config self-check refuses a view with no enabled VIEW_SOURCES (line 881). The catalog's own dependency edges from harvest (dependency_edges in ApplyStats, apply.ts:51, written at apply.ts:489) are recorded, but whether they feed this canvas was not measured here.

8. REST bridge

app/api/app/data_catalog/[verb]/route.ts is the single generic dispatcher (header lines 6-17): verb table from _bridge/registry.ts ← DATA_CATALOG_VERB_METHODS (wire.ts:411-419); door floor per verb from COMMAND_AUTH.data_catalog + tier config; args parsed by the same Zod union; executed by the same dispatchDataCatalog inside a request scope carrying the REAL principal (resolveCatalogPrincipal; the DC-2 fix that removed a constant isAdmin: true, lines 18-27). Success: 200 { data }. Refusal: RFC 9457 problem+json (lib/server/problem.ts) whose code is the registry key and status from REFUSAL_HTTP_STATUS; door refusals are explicit 403 with their own type. Usage metered as web:data_catalog.<verb>.

GET verbs: help, provider_list, provider_get, crawler_list, assets_list, asset_get, columns_list, columns_search, objects_search, profile_get, terms_list, terms_grouped_list, bindings_list, binding_get, flow_summary. POST verbs: every writer (provider_create/update/test/retire, crawler_create, scan_run, terms_materialize, bind_inbound, unbind, remaster, asset_tag_set/remove, capture_payload, asset_register_enricher).

9. Common errors and their fix hints

From the ONE registry lib/mcp/dataCatalogErrors.ts (envelope {ok:false, error:{code, message, retryable, fix:{tool, command?, args, why}}}, line 9). The registry declares 58 codes; the table below covers all 58. Hints are the registry's own fix.why, condensed.

10. Trust notes

• Everything above was read from source; nothing was executed against the live database, so counts (e.g. glossary size "18,223 entries" quoted in wire.ts comments) are as of the comment, not measured now.
• Unmeasured here: the full QUALITY_UNAVAILABLE_REASONS vocabulary, the per-kind governance-source table (help), and whether harvest dependency_edges reach the lineage canvas.
• Standards applied on this surface: ISO 8601 durations / RFC 5545 RRULE for cadences, RFC 3339 for flow_summary bounds, RFC 9457 for REST refusals (MCP surface uses the compact envelope), RFC 9110 §15 for http_response_code, GDPR Art. 5/25 and HIPAA 164.502(b) for the sample-value default.

Measured against the release named in the text. Raw Markdown: /ontology/documentation/data-catalog.md. This page is also served as raw Markdown at the same URL with a .md suffix, for agents and search tools that prefer plain text. Every capability statement cites the source file and line it was measured from.