A data engineer wrote a notebook six months ago that reads from gold.finance.revenue_summary and writes to gold.reporting.quarterly_kpis. A colleague now opens the quarterly_kpis table in Catalog Explorer to trace where the data originally came from. They also want to understand whether column-level lineage is captured. Which combination of statements CORRECTLY describes Unity Catalog's lineage behavior for this scenario?
Answer & explanation
Correct answer: E
WHY E: Unity Catalog lineage needs no opt-in or configuration: it is captured at runtime whenever queries run on Databricks against Unity Catalog-registered tables through Spark DataFrame or Databricks SQL interfaces. The documentation says lineage is 'supported for all languages' and captured 'down to the column level', and clicking a column in the graph shows the upstream columns it was derived from. Opening the table in Catalog Explorer, choosing the Lineage tab and 'See Lineage Graph' is the documented procedure. Lineage data is retained for a rolling one-year window, so the link between revenue_summary and quarterly_kpis from a run six months ago is still visible.
WHY NOT A: Unity Catalog does not require lineage enrollment with ALTER TABLE ... ENABLE LINEAGE; capture happens for all qualified workloads without setup. Column-level lineage is supported for Delta tables and has no format-based restriction to Iceberg.
WHY NOT C: The documentation states lineage is 'supported for all languages' and captured via 'Spark DataFrame (for example, Spark SQL functions that return a DataFrame) or Databricks SQL interfaces such as notebooks or the SQL query editor'. Notebooks using DataFrames or Spark SQL are covered; the classic versus serverless compute distinction does not exclude notebook lineage.
WHY NOT D: Lineage is 'aggregated across all workspaces attached to a Unity Catalog metastore', so a run from another workspace on the same metastore shows up in the colleague's view. Column-level lineage is inferred from query plans and needs no manual annotation by a data steward.
WHY NOT B: Lineage data is retained for one year (a rolling window), not indefinitely, and there is no ALTER TABLE ... DROP LINEAGE command. The documentation also lists cases where column-level lineage is not captured: UDFs (which can obscure column mapping), path-based table references such as select * from delta.'s3://...', RDD-based operations, global temp views, and checkpointed datasets.