Pull from Databricks Unity Catalog

kiri catalog push sends your governance out to Unity. kiri catalog pull reads reality back in: the schemas Unity actually holds, the lineage Databricks actually observed, and the usage signals that tell you what is really being queried. Pull is how Kirimana keeps its contract-first view honest against the live platform — it does not overwrite your contracts, it surfaces where the platform and the contracts have diverged so a human can decide what to do.

Pull is read-only and one-directional. It never mutates Unity and never silently mutates your contracts; it emits an asset stream that downstream commands reconcile against what the contracts declare.

Pulling the catalog

Pull emits NDJSON — one asset per line — that you can inspect or chain straight into the local catalog:

# Pull metadata from a Unity catalog to stdout (one Asset per line).
kiri catalog pull --target unity --catalog main

# Restrict to specific schemas.
kiri catalog pull --target unity --catalog main \
  --schema sales --schema finance

# Chain the stream directly into the local catalog DB.
kiri catalog pull --target unity --catalog main \
  | kiri catalog import --from-ndjson -

By default every schema in the catalog is pulled except information_schema. --no-columns does a faster inventory-only pull that skips column-level metadata; --no-tags skips the table-level tag round-trip. Piping kiri catalog pull into kiri catalog import --from-ndjson - is the designed pattern — it loads the observed state into the same local catalog your contracts are compared against.

Schema-drift detection

Once the observed schema is in the local catalog, you can compare it against what the contracts declare. kiri catalog verify diffs the live catalog annotations against the local catalog and reports typed findings:

• missing — the contract expects an annotation Unity does not have.
• stale — the annotation in Unity no longer matches the contract.
• extra — Unity carries something the contract does not account for (a column added directly in the platform, for instance).

kiri catalog verify --target unity --fail-on all --output report.json

--fail-on takes a closed menu — missing, stale, extra, or all — and exits non-zero when a finding of that kind is present, so you can gate a pipeline on the catalog and the contracts staying in step. The JSON report is machine-readable for feeding into a dashboard or ticket.

Observed lineage

Databricks records the lineage it actually observes — which table fed which — in Unity’s system column-lineage view. Kirimana reads those observed edges and reconciles them against the lineage your contracts declare. This is a genuinely useful check: a contract can claim a silver table derives from a bronze source, but only the platform knows whether that pipeline actually ran that way.

The reconciliation is careful about identity. Declared edges are keyed by logical contract URNs; Unity’s observed edges are keyed by physical catalog.schema.table names. Comparing them naively produces almost entirely false drift, because they live in different identity spaces. Kirimana bridges every observed physical endpoint back to the contract that governs it — using the same physical-naming rules the Databricks adapter and the catalog importer use — and only then computes the diff. The result is existence-grade: it reconciles the lineage fact (“this table feeds that table”), because Unity’s observed lineage carries no semantic edge kind to assert anything finer.

The drift report is bounded by construction. An enterprise warehouse can drift in hundreds of edges, so the report caps its findings and carries an honest truncated signal rather than returning an oversized blob — you get “drift capped at N”, not a wall of noise.

Downstream usage signals

The observed side of the pull is not just structure. Because Unity knows what actually queried what, the observed edges double as usage signals — which downstream tables and consumers really depend on an asset. That answers questions the contracts alone cannot: whether a column flagged for deprecation is still being read, or whether a “critical” table has any live consumers at all. Usage grounds a governance decision in observed behaviour instead of assumption.

How pull feeds reconciliation

Pull is the input stage of a reconcile loop, not the whole loop:

1. Pull the live schema and observed lineage from Unity (kiri catalog pull), loading it into the local catalog.
2. Detect drift between observed and declared — schema drift via kiri catalog verify, lineage drift via the declared-vs-observed reconciliation engine.
3. Explain, do not auto-apply. Kirimana explains the drift and may propose remediation; it never rewrites your contracts on its own. A human reviews the drift and lands the fix through a normal contract PR — which then goes through the PR-time governance gates like any other change.

That last principle is the whole point of the direction. Push makes the platform reflect the contracts; pull tells you where the platform has drifted from them; and the fix always flows back through git and review, never as a silent catalog-to-contract overwrite. The contract corpus stays the single source of truth, and the platform stays observably in step with it.

For the outbound direction — projecting owners, classifications, review state, and lineage into Unity — see Push to Databricks Unity Catalog.