Debug a failed apply

An apply that fails is not a mystery to be reverse-engineered from Databricks. Every apply — success or failure — emits an audit line carrying a trace id, and that id is the thread you pull to diagnose what happened. This page walks the loop: read the audit line, classify the failure into one of a few known shapes, fix the cause, and re-run.

Start with kiri doctor

Before you classify anything by hand, run the project health punch-list:

kiri doctor --target test

kiri doctor composes the checks Kirimana already ships — contract structural and semantic validation, modeling-technique consistency, layer-contract validation, source validation, fixture presence, and, when a reachable --target is given, the adapter health probe — into one ranked report. Findings sort by severity then layer (bronze → silver → gold), and each carries a fix hint and, where the fix is a Kirimana command, a copy-pasteable one. A great deal of what would otherwise surface as a failed apply — an invalid contract, a technique mismatch, an unreachable target — is caught here first, with the fix already named.

Two properties make it CI-friendly as well as interactive:

kiri doctor --target test --format json --strict

--format json emits the typed report for a pipeline to consume, and the exit code is non-zero on any error-severity finding (with --strict promoting warnings to failures too). Checks that need a target you haven’t supplied — or that can’t reach the one you did — are reported as skipped, never failed: doctor fails closed on real findings and fails open on absent connectivity. Scope a run with --only or --skip against the check registry when you want a narrower pass.

If kiri doctor comes back clean and the apply still failed, the cause is environmental rather than in your project — work the audit line below.

Start at the audit line

Kirimana’s apply-family verbs print a bracketed audit line as their last output, of the form:

--- <what-ran> (trace_id=<id>) ---

The trace_id is stable across the whole operation. It is what you quote when you ask for help, what you grep the audit store for, and what ties the CLI output to the workspace-side record. Copy it before you do anything else — a failed apply that scrolled off your terminal is still recoverable by its trace id.

The audit store backs kiri release history and the approval verbs, so a promotion that failed leaves a record you can inspect after the fact:

kiri release history --env test --output json

Classify the failure

Most failed applies fall into one of four classes. Identify the class first; the fix follows from it.

Auth — the credential is missing, expired, or revoked

Symptom: the failure names authentication, a 401, or a vault ref that would not resolve. The apply never reached the point of changing anything.

The execution identity is the Databricks service principal, and its token lives in the vault, referenced from kiri.yml. If it has expired or been rotated at the platform without the vault being updated, every apply fails at the door. Confirm the wiring:

kiri databricks health --target test

The vault-scope probe verifies every ${vault:…} ref in kiri.yml resolves; a FAIL there is your auth problem. Rotate by re-setting the value under the same <id>:<key> and re-run health (see the vault docs).

Permissions — the identity is real but under-granted

Symptom: a 403, or a specific Unity Catalog / Workflows operation denied. The service principal authenticated but lacks a grant the apply needs — CREATE SCHEMA, CREATE TABLE, MODIFY, or Can Manage on Workflows.

kiri databricks health probes each of these independently and, on any FAIL, prints the exact GRANT statement that fixes it. Take that statement to whoever owns the workspace. This is a platform-side fix, not a code change — do not touch contracts to work around a missing grant.

Drift — the workspace no longer matches the plan

Symptom: the apply’s expectations about existing objects don’t hold — a table someone changed by hand, a schema that vanished, a shape the plan didn’t anticipate.

Re-plan against the environment to see the true diff before you do anything else:

kiri release plan --to test

If the diff shows changes you didn’t intend, something out-of-band edited the workspace. The durable fix is to bring the change into the repo (so the next apply is authoritative) rather than to keep patching by hand. For migration cutovers, kiri migrate verify --target test re-runs the pre-deploy infrastructure checks and refuses to pass unless catalog, schemas, external location, and warehouse are all present and correct.

Lint — the contracts themselves are invalid

Symptom: the apply refuses to start, citing a contract or source that fails validation. This is the cheapest class to fix because it never touched the workspace.

Lint is a code-level concern — a malformed contract, a raw secret in YAML where a ${vault:…} ref belongs, a layer-policy violation. Fix it in the repo, re-run your validation locally, commit, and re-plan. Migration projects have a dedicated gate, kiri migrate lint-models, which applies the medallion layer-policy across generated dbt models.

Retry semantics

Apply is designed to be safe to re-run. Once you have removed the cause — refreshed a token, added a grant, corrected a contract — promote the same SHA again:

kiri release apply --to test

Because promotion is keyed on the git SHA, re-running with the same commit checked out re-materialises the environment to that exact state. There is no half-applied limbo to clean up first: fix the cause, re-apply the SHA, and confirm with kiri release status. Transient vault reads (a 5xx or a network blip from the secret backend) are retried inside the apply under a bounded policy, so a momentary hiccup will not by itself fail the run — a failure that surfaces to you has already exhausted those retries.

On kiri explain

kiri explain is a reserved verb whose full behaviour — tracing a generated line back to the exact contract field, hook, or template override that produced it — is not yet implemented in v1.0.0-beta.1. For debugging today, the audit line’s trace id plus kiri release plan and kiri databricks health are the supported tools. Do not rely on kiri explain output as authoritative until it ships.