A solutions architect is designing a new data platform on Databricks. She has three distinct requirements: (1) ingest streaming events from Apache Kafka and declaratively apply multi-hop transformations before writing clean data to Delta tables, (2) enforce fine-grained column-level access controls and capture data lineage across all data assets, and (3) schedule and orchestrate a nightly batch aggregation job that depends on the successful completion of the streaming pipeline. Which combination of Databricks platform components BEST maps to these three requirements, in order?
Show answer & explanation
Correct answer: B
WHY B is correct: Each requirement maps precisely to the described Databricks component. (1) Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables) is expressly designed as a declarative framework for batch and streaming data pipelines using Python/SQL. It includes connectors for sources such as Apache Kafka, Amazon Kinesis, and Google Pub/Sub, and supports multi-hop transformations through Bronze, Silver, and Gold layer patterns. (2) Unity Catalog is the unified governance layer that enforces column-level masking and row-level security through dynamic views and table policies, and it automatically tracks column-level lineage — capturing which upstream tables, views, and notebooks produced each output column. (3) Lakeflow Jobs is the Databricks workflow orchestration service that schedules and runs tasks (notebooks, pipelines, SQL queries, Python scripts) with dependency chains, cron triggers, and retry logic. WHY NOT A: Databricks SQL is designed for BI analytics queries and dashboard workloads, not for building stateful streaming ingestion from Kafka. SQL Warehouses are not designed to act as long-running Kafka consumers. Delta Lake is a storage format with transaction support — it does not contain an access policy enforcement engine; that role belongs to Unity Catalog. Mosaic AI is the AI/ML platform, not a workflow scheduling service. WHY NOT C: Lakeflow Jobs is an orchestration service for scheduling tasks — it does not act as a Kafka streaming consumer or apply streaming transformations. Mosaic AI's Feature Store manages ML features, not general-purpose data table access controls. Delta Lake does not contain a built-in CRON scheduler; scheduling is handled by Lakeflow Jobs. WHY NOT D: Unity Catalog does not contain Kafka connectors or streaming ingestion capabilities — it is a governance layer. Databricks SQL does not provide visual column masking editors that generate lineage; column masking is configured through Unity Catalog policies. Lakeflow SDP (Delta Live Tables) is a pipeline framework, not a job scheduler for triggering batch compaction runs. WHY NOT E: Delta Lake is a storage layer format (providing ACID transactions, time travel, and schema enforcement) — it does not contain Kafka consumer clients. The ingestion framework with Kafka connectivity is Lakeflow SDP. Lakeflow Jobs applies cluster-level access modes but does not enforce column-level security — that is Unity Catalog's role. Unity Catalog is a governance system, not a pipeline scheduler.