A solutions architect is designing a new data platform on Databricks. She has three distinct requirements: (1) ingest streaming events from Apache Kafka and declaratively apply multi-hop transformations before writing clean data to Delta tables, (2) enforce fine-grained column-level access controls and capture data lineage across all data assets, and (3) schedule and orchestrate a nightly batch aggregation job that depends on the successful completion of the streaming pipeline. Which combination of Databricks platform components BEST maps to these three requirements, in order?
Answer & explanation
Correct answer: B
WHY B: Each requirement maps to the described component. (1) Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables) is a declarative framework for batch and streaming pipelines in Python/SQL. It includes connectors for sources such as Apache Kafka, Amazon Kinesis, and Google Pub/Sub, and supports multi-hop transformations through Bronze, Silver, and Gold layer patterns. (2) Unity Catalog is the unified governance layer that enforces column-level masking and row-level security through dynamic views and table policies, and it tracks column-level lineage, capturing which upstream tables, views, and notebooks produced each output column. (3) Lakeflow Jobs is the workflow orchestration service that schedules and runs tasks (notebooks, pipelines, SQL queries, Python scripts) with dependency chains, cron triggers, and retry logic.
WHY NOT A: Databricks SQL is designed for BI analytics queries and dashboards, not for stateful streaming ingestion from Kafka; SQL warehouses are not long-running Kafka consumers. Delta Lake is a storage format with transaction support and has no access policy enforcement engine; that role belongs to Unity Catalog. Mosaic AI is the AI/ML platform, not a workflow scheduling service.
WHY NOT C: Lakeflow Jobs is an orchestration service for scheduling tasks; it does not act as a Kafka streaming consumer or apply streaming transformations. Mosaic AI's Feature Store manages ML features, not general-purpose table access controls. Delta Lake has no built-in cron scheduler; scheduling is handled by Lakeflow Jobs.
WHY NOT D: Unity Catalog is a governance layer and has no Kafka connectors or streaming ingestion capabilities. Databricks SQL does not provide a visual column masking editor that generates lineage; column masking is configured through Unity Catalog policies. Lakeflow Spark Declarative Pipelines is a pipeline framework, not a job scheduler for triggering batch compaction runs.
WHY NOT E: Delta Lake is a storage layer format (ACID transactions, time travel, schema enforcement) and has no Kafka consumer clients; the ingestion framework with Kafka connectivity is Lakeflow Spark Declarative Pipelines. Lakeflow Jobs applies cluster-level access modes but does not enforce column-level security, which is Unity Catalog's role. Unity Catalog is a governance system, not a pipeline scheduler.