Databricks Intelligence Platform: Databricks Data Engineer Associate Practice Questions

A data engineering team at a media company is tasked with building a production-grade streaming data pipeline that ingests clickstream events from Apache Kafka, enforces data quality rules at the Silver layer (e.g., rejecting records with null user_id or negative session_duration), fans out to multiple Gold layer tables with different aggregation windows, and must automatically reprocess downstream Gold tables whenever upstream Silver table logic is updated. The pipeline must run in both continuous streaming mode during business hours and scheduled batch mode on weekends, support built-in lineage visualization to satisfy a data governance audit, and minimize the operational burden on the engineering team (no manual cluster management or Spark job orchestration code). A senior data engineer evaluates four compute options and asks which is the best architectural fit. Which Databricks service is the most operationally appropriate choice for all stated requirements?

1. ALakeflow Spark Declarative Pipelines (the rebrand of Delta Live Tables as of late 2025) is the correct service: it natively supports both streaming and batch execution modes in the same pipeline definition, enforces data quality via @dlt.expect / @dlt.expect_or_drop decorators or CONSTRAINT clauses in SQL, automatically infers and manages inter-table dependencies, propagates upstream changes to downstream tables, integrates lineage tracking directly into Unity Catalog, and removes the need for manual cluster lifecycle management.
2. BA Classic SQL Warehouse provisioned with maximum auto-scaling and enhanced Photon acceleration would best satisfy all requirements: it can ingest Kafka streams via a native Kafka SQL connector available in Databricks SQL, enforce NOT NULL column constraints on Silver tables using Delta table CHECK constraints, and fan out to Gold tables through a set of scheduled SQL tasks defined in Databricks Workflows, with lineage automatically captured in Unity Catalog for every SQL statement executed through the warehouse.
3. CA long-lived all-purpose cluster running a single Spark Structured Streaming application written in PySpark would be the best choice because it can maintain Kafka consumer offsets in a checkpoint directory, and Databricks notebooks provide a built-in streaming monitoring UI that shows input rates and processing times per micro-batch, with data quality enforced using DataFrame filter() operations before writing to Silver tables and manual notebook scheduling handling the weekend batch mode transition.
4. DA multi-task Databricks Workflow using one job cluster per task would provide the greatest operational flexibility: the Kafka ingestion task runs a PySpark Structured Streaming job, Silver transformation tasks run independently on separate job clusters triggered by an upstream dependency defined in the DAG, Gold aggregation tasks are parameterized notebooks scheduled with different cron windows for business-hour and weekend modes, and data quality is validated post-hoc by a dedicated validation task that queries each table after every write and sends alerts via webhook if row counts fall below expected thresholds.

A large retail organization currently runs a two-tier data architecture: a cloud data lake (Amazon S3 with Parquet files) for historical raw data storage, and a proprietary cloud data warehouse for curated reporting and BI. The data science team must copy data from the warehouse back into S3 when training machine learning models, creating significant data duplication and pipeline complexity. The platform is also struggling with inconsistent access control: governance policies in the warehouse do not extend to the raw lake, leading to compliance audit findings. The CTO is evaluating a migration to the Databricks Data Intelligence Platform. An architect is asked to explain how adopting the platform and Unity Catalog specifically addresses the organization's four concerns: (1) eliminating data duplication between systems, (2) supporting both SQL analytics and ML on the same data, (3) implementing fine-grained access control with full auditing and lineage, and (4) reducing total cost of ownership. Which statement best and most completely addresses all four concerns?

1. AMigrating to the Databricks Data Intelligence Platform reduces licensing costs by converting all existing proprietary warehouse tables into Apache Parquet files stored in cloud object storage, while still requiring organizations to retain their existing relational data warehouse for production SQL analytics and BI workloads, with Unity Catalog providing one-way metadata synchronization from the warehouse to Databricks to keep governance policies loosely aligned across both systems.
2. BDatabricks resolves the governance gap by deploying Unity Catalog as a sidecar service alongside the organization's existing proprietary data warehouse, synchronizing permission models bidirectionally every 15 minutes so that access control policies remain consistent across both platforms, while data scientists continue to copy warehouse data to the lake for ML training because Databricks Spark clusters cannot directly query live warehouse tables without first materializing them as Delta tables.
3. CThe Databricks Data Intelligence Platform addresses data duplication by maintaining a proprietary binary columnar format that is 40% more compact than Parquet, allowing both the data warehouse team and the data science team to store copies of the same dataset at lower cost; Unity Catalog enforces column-level security within Databricks notebooks but does not extend governance to external BI tools unless those tools natively integrate with the Databricks Unity Catalog REST API.
4. DThe Databricks Data Intelligence Platform stores all data in open Delta Lake format on the organization's own cloud storage, enabling SQL analytics, machine learning, and streaming workloads to operate on the same physical dataset without data movement or duplication; Unity Catalog provides fine-grained access control, comprehensive audit logging, and cross-workspace data lineage tracking; and consolidating onto a single platform versus maintaining two separate systems reduces infrastructure and licensing costs.

A financial analytics company has a team of 60 business analysts who run ad-hoc SQL queries throughout the business day against curated Delta tables in Unity Catalog. Query patterns are highly variable: some analysts submit simple lookups that finish in seconds, while others trigger complex aggregations that can run for several minutes. The infrastructure team is currently managing three Classic SQL Warehouses (size Large, 4 clusters each) and struggles with over-provisioning during low-traffic hours and under-provisioning during morning peaks. The platform team is evaluating whether to migrate to Serverless SQL Warehouses. A stakeholder asks for a precise comparison of how Serverless SQL Warehouses handle scaling and infrastructure management compared to Classic SQL Warehouses. Which statement is most accurate?

1. AServerless SQL Warehouses still require the data engineering team to configure a minimum and maximum cluster count, define auto-suspend thresholds, and select a node instance type for the underlying virtual machines, but Databricks takes over OS-level patching and Spark version upgrades, reducing administrative overhead while leaving capacity planning decisions entirely with the platform team.
2. BServerless SQL Warehouses execute SQL queries by dynamically borrowing idle compute capacity from all-purpose clusters already running in the same Databricks workspace, which means the analytics team pays nothing incremental for query execution as long as interactive clusters are active, but the data engineering team must maintain at least two all-purpose clusters in the workspace at all times to guarantee warehouse availability during business hours.
3. CServerless SQL Warehouses scale compute automatically based on active query concurrency without requiring cluster size or node configuration from the user, are fully managed by Databricks (zero infrastructure decisions), start in under two seconds due to pre-warmed compute pools, and bill only for the duration of actual query execution — making them optimal for the variable concurrent SQL workload described.
4. DServerless SQL Warehouses are architecturally optimized exclusively for scheduled overnight batch SQL workloads because they operate on a shared multi-tenant serverless compute pool, and during business hours when many organizations simultaneously submit interactive queries to this shared pool, the resulting resource contention introduces query queue latencies averaging 30–90 seconds that make them unsuitable for responsive ad-hoc analytics scenarios.

A data platform team supports three distinct workloads running on Databricks: (1) A data scientist who explores a newly delivered raw dataset interactively in a Databricks notebook, iterating rapidly on transformations and running ad-hoc visualizations over a session lasting several hours. (2) A nightly automated ETL job that reads from Bronze Delta tables, applies heavy PySpark aggregations across 5 TB of data, and writes results to Gold layer tables. The job must minimize cost and run in a fully isolated compute environment. (3) A group of 30 business analysts using a BI tool connected via JDBC to run concurrent SQL queries against Gold layer tables in Unity Catalog throughout the workday. The platform lead wants to ensure each workload is matched to the most operationally appropriate and cost-efficient Databricks compute type. Which mapping correctly aligns each workload with its optimal compute resource?

1. AWorkload 1 → SQL Warehouse (Serverless), because Serverless SQL Warehouses support notebook-based interactive exploration; Workload 2 → All-purpose cluster with auto-termination set to 30 minutes, because the cost savings from auto-termination offset the expense of a persistent all-purpose cluster; Workload 3 → Job cluster, because job clusters support multi-user concurrent SQL via Databricks SQL endpoints exposed through the cluster's built-in HTTP server.
2. BWorkload 1 → All-purpose cluster, for interactive, session-based notebook exploration with shared libraries and iterative development; Workload 2 → Job cluster (or Serverless job compute), for isolated, cost-efficient automated batch processing that terminates when the job completes; Workload 3 → SQL Warehouse (Classic or Serverless), optimized for concurrent multi-user SQL analytics via JDBC/ODBC with auto-scaling and query isolation.
3. CWorkload 1 → Lakeflow Spark Declarative Pipeline cluster, to leverage built-in data quality expectations and lineage tracking during exploratory analysis; Workload 2 → SQL Warehouse, because SQL Warehouses can execute PySpark transformation logic submitted via the Databricks SQL API; Workload 3 → All-purpose cluster in high-concurrency mode, which provides the best performance for concurrent BI tool connections through the legacy cluster-based SQL endpoint.
4. DWorkload 1 → Job cluster with all-purpose cluster configuration flag set to true, enabling it to serve as a dual-purpose resource for both interactive notebook use and scheduled jobs, which reduces total cluster count and cost; Workload 2 → All-purpose cluster with a 240-minute auto-termination timeout configured to match the maximum expected job duration; Workload 3 → Job cluster with the multi-user isolation mode enabled and Photon acceleration turned on to handle the concurrent analyst queries routed through the JDBC driver.