Productionizing Data Pipelines: Databricks Data Engineer Associate Practice Questions

A data engineering team wants to ensure their Databricks Asset Bundle deployment process follows CI/CD best practices. They are setting up a GitHub Actions pipeline. In the CI step (on pull request), they want to validate that the bundle configuration is syntactically correct and all referenced files exist, WITHOUT actually deploying any resources to the Databricks workspace. In the CD step (on merge to main), they want to deploy to production. Which pair of databricks bundle commands should be used for the CI and CD steps respectively?

1. ACI: databricks bundle plan -t prod to generate a deployment plan showing what would change, similar to Terraform plan; CD: databricks bundle apply -t prod to execute the plan and deploy resources, similar to Terraform apply.
2. BCI: databricks bundle validate to check the bundle configuration for syntax errors, schema compliance, and file reference validity without connecting to or modifying any Databricks workspace; CD: databricks bundle deploy -t prod to authenticate, upload artifacts, and create or update all defined workspace resources in the production target.
3. CCI: databricks bundle test --dry-run -t prod to simulate a full deployment against a sandbox environment and report any resources that would fail to provision; CD: databricks bundle deploy -t prod --confirm to deploy with an explicit confirmation flag required for production targets.
4. DCI: databricks bundle lint -t prod to run static analysis checks on all YAML and Python files in the bundle, including checking for deprecated Databricks Runtime versions and unsupported cluster configurations; CD: databricks bundle release -t prod to tag the bundle version in Git and trigger the production deployment workflow.
5. ECI: databricks bundle check -t prod to verify that all Databricks workspace permissions required by the bundle are correctly configured and that the service principal used for deployment has sufficient access; CD: databricks bundle push -t prod to upload all bundle artifacts to the Databricks workspace file system before a separate bundle activate command makes the resources live.

A data engineer is inspecting the Spark UI for a job that performs a large aggregation. In the 'SQL / DataFrame' tab, the physical plan shows the following operators in sequence: Exchange hashpartitioning → Sort → HashAggregate. The job is taking much longer than expected. The engineer enables Adaptive Query Execution (AQE). Which AQE optimization is MOST relevant to this aggregation pattern, and what specific improvement does it make?

1. AAQE's Dynamic Partition Pruning (DPP) is most relevant here. DPP intercepts the Exchange operator and prunes shuffle partitions that do not contain data matching the aggregation's GROUP BY keys, reducing the amount of data sent across the network before the Sort and HashAggregate phases.
2. BAQE's Coalescing of Shuffle Partitions is most relevant here. After the shuffle (Exchange hashpartitioning), AQE examines the actual sizes of the shuffle map output partitions at runtime and merges small partitions together before they are read by the Sort and HashAggregate operators, reducing the overhead of processing thousands of tiny partitions and minimizing task scheduling overhead.
3. CAQE's Skew Join Optimization is most relevant here. It detects that one or more HashAggregate partitions are significantly larger than others and automatically splits the oversized aggregate partition into smaller sub-partitions, each handled by a separate task, effectively parallelizing the final aggregation phase.
4. DAQE's Broadcast Join Conversion is most relevant here. When AQE determines at runtime that one side of the aggregation's input data is smaller than the broadcast threshold after the Exchange operator, it converts the HashAggregate into a BroadcastHashAggregate, eliminating the Sort operator from the physical plan entirely.
5. EAQE's Runtime Filter Injection is most relevant here. AQE analyzes the GROUP BY column statistics after the Exchange phase and injects a Bloom filter into the HashAggregate operator to skip processing of rows whose key values have already been aggregated in a prior micro-batch, reducing the final aggregate computation time.

A data engineer is reviewing a Databricks Asset Bundle project that another team member created. The databricks.yml file contains the following snippet: ``yaml targets: dev: mode: development workspace: host: https://adb-dev-1234.azuredatabricks.net variables: catalog_name: dev_catalog prod: mode: production workspace: host: https://adb-prod-5678.azuredatabricks.net variables: catalog_name: prod_catalog ` The engineer wants to deploy to the production environment. Which databricks bundle CLI command is correct, and what behavior will the mode: production` setting enforce?

1. Adatabricks bundle deploy --target=production is correct, but mode: production has no functional enforcement effect—it is only a metadata label used for documentation purposes within the bundle configuration and does not change any deployment behavior.
2. Bdatabricks bundle deploy -t prod is the correct command. The mode: production setting enforces behaviors such as preventing concurrent runs of the same job by default, tagging all deployed resources with the bundle name for auditability, and disabling 'development mode' protections like resource name prefixing—ensuring production resources have clean, unprefixed names.
3. Cdatabricks bundle publish --env prod is the correct command, as DAB uses the publish subcommand for production targets to distinguish them from lower-environment deploy operations, and mode: production automatically triggers a blue-green deployment strategy to avoid downtime.
4. Ddatabricks bundle deploy --workspace prod is the correct command, as the --workspace flag is used to specify the target name in the YAML configuration, and mode: production automatically locks the bundle configuration to prevent any further edits until the deployment is confirmed as healthy.
5. Edatabricks bundle deploy --stage prod is the correct command, using the --stage flag to reference the named target, and mode: production automatically enables Delta Lake data retention policies and Z-ordering on all tables written by the deployed pipeline.

During a Spark UI analysis, a data engineer observes that a job's 'Executors' tab shows several executors with very high 'GC Time' as a percentage of 'Task Time' (over 25%), while other executors show normal GC time (under 5%). All executors have identical hardware configurations. What is the MOST likely cause of this uneven GC pressure, and what investigation step should the engineer take next in the Spark UI?

1. AHigh GC time on specific executors indicates that those executors are running tasks that involve Python UDF execution, which creates excessive JVM object allocation due to ser/deserialization between the JVM and Python worker process. The next step is to navigate to the 'Environment' tab in the Spark UI to confirm that spark.python.worker.reuse=false is set, which would cause a new Python process to be spawned for every task.
2. BHigh GC time on specific executors most likely indicates data skew—those executors are receiving and processing disproportionately large partitions, causing their JVM heap to fill rapidly with live objects during processing, triggering frequent garbage collection cycles. The next investigation step is to examine the 'Stages' tab, click on the relevant stage, and inspect the 'Task Metrics' distribution (specifically the 'Input Size / Records' and 'Shuffle Read Size' columns) to identify which tasks on those executors are receiving oversized partitions.
3. CHigh GC time on specific executors indicates that those executors are co-located on the same physical cloud VM and are sharing CPU resources, causing the JVM's GC threads to be starved of CPU time. The next investigation step is to navigate to the Databricks cluster's 'Metrics' tab (Ganglia) and inspect per-node CPU utilization to identify VM co-location conflicts.
4. DHigh GC time on specific executors indicates a JVM version mismatch—some executors are running a JVM version that uses a less efficient garbage collector algorithm (e.g., Serial GC instead of G1GC). The next investigation step is to check the 'Environment' tab in the Spark UI for each executor's java.vm.version property to identify executors running older JVM builds.
5. EHigh GC time on specific executors indicates that the Spark block manager on those executors is caching too many RDD partitions in the JVM heap, causing heap exhaustion and triggering GC to evict cached blocks. The next investigation step is to navigate to the 'Storage' tab in the Spark UI and examine the 'RDD Storage Info' section to identify which cached RDDs are consuming the most memory on the affected executors.

Given a scenario where a data engineer is evaluating serverless compute for a production Delta Live Tables (DLT) pipeline, which of the following statements about Databricks Serverless compute is ACCURATE?

1. AServerless compute for DLT pipelines provisions resources in the customer's own cloud account VPC, giving the customer full network visibility and control over egress traffic while Databricks manages the orchestration layer.
2. BServerless compute requires the customer to pre-purchase a reserved instance commitment with their cloud provider to access the serverless pool, after which Databricks manages the allocation of those reserved instances automatically.
3. CServerless compute runs in Databricks-managed cloud infrastructure, starts within seconds, automatically scales to match pipeline demands, requires no cluster configuration by the user, and bills only for the compute actually consumed—making it ideal for workloads with unpredictable or bursty resource needs.
4. DServerless compute for DLT is only available for SQL-based pipelines and does not support Python-defined Delta Live Tables expectations or Python UDF decorators within the pipeline definition.
5. EServerless compute uses a shared multi-tenant execution environment where multiple customers' pipeline code runs on the same worker nodes simultaneously, which reduces cost but requires customers to avoid storing secrets or PII in pipeline variables.

A data engineer notices that a Spark job processing 500 million records is taking 45 minutes to complete, significantly longer than expected. They open the Spark UI to investigate. In the 'Stages' tab, they observe one stage with 200 tasks where 199 tasks completed in under 10 seconds each, but one task took 38 minutes. The data being processed is partitioned by a country_code column. What does this pattern MOST LIKELY indicate, and what is the recommended remediation?

1. AThe pattern indicates that the Spark driver node ran out of memory while collecting results from the 200 executors, causing a garbage collection pause on the driver that delayed the final task completion acknowledgment. The recommended remediation is to increase the driver node memory and enable off-heap storage for the result set.
2. BThe pattern indicates data skew, where a single partition (likely one dominant country_code value such as 'US') contains a disproportionately large share of the total data. The slow task is processing this oversized partition alone. The recommended remediation includes techniques such as salting the skewed key, using skewHint in Spark SQL, or applying AQE's skew join optimization (spark.sql.adaptive.skewJoin.enabled=true).
3. CThe pattern indicates that the Spark scheduler assigned all 200 tasks to a single executor due to a YARN resource manager misconfiguration, serializing what should be parallel execution. The recommended remediation is to reconfigure the YARN node manager memory settings and increase spark.executor.instances to force true parallelism.
4. DThe pattern indicates that the shuffle write phase produced excessively large shuffle files for one partition, causing the Spark external shuffle service to throttle that partition's data transfer to comply with cloud storage bandwidth quotas. The recommended remediation is to increase the number of shuffle partitions via spark.sql.shuffle.partitions to distribute the shuffle write load.
5. EThe pattern indicates a broadcast join gone wrong, where the broadcast variable for the lookup table exceeded the 8 MB default threshold mid-execution, causing Spark to fall back to a sort-merge join strategy for one partition while completing broadcast joins for all others, resulting in asymmetric task completion times.

A data engineer is tasked with deploying a production Databricks Workflow that orchestrates four tasks: (1) a Python notebook for data ingestion, (2) a Delta Live Tables pipeline for transformation, (3) a SQL notebook for data quality checks, and (4) a Python script for sending an email notification on success or failure. The team uses Databricks Asset Bundles. The engineer also needs to configure the workflow so that Task 4 (notification) always runs, even if Tasks 2 or 3 fail. In the DAB job resource YAML, which task dependency configuration correctly implements the 'always run notification' requirement?

1. ASet Task 4's depends_on to reference all three upstream tasks with outcome: success for each dependency. This instructs Databricks to run Task 4 only after confirming all upstream tasks completed successfully, which is the only supported dependency outcome type in DAB YAML configurations.
2. BSet Task 4's depends_on to list Tasks 1, 2, and 3 without any outcome filter (or with the default outcome: success). Then, in the Workflows UI after deployment, manually edit Task 4's dependency settings to add an 'On failure' trigger path, since DAB YAML does not support conditional dependency outcomes and this configuration must be done through the UI.
3. CDefine Task 4 as a completely independent task with no depends_on configuration, and instead configure it with a separate trigger schedule that fires 10 minutes after the main workflow's expected completion time, ensuring it runs regardless of upstream task outcomes.
4. DIn the DAB job resource YAML, configure Task 4's depends_on block to reference each upstream task with outcome: success_or_failed for Tasks 2 and 3 (to handle both success and failure), and outcome: success for Task 1. This combination of outcome types within a single task's dependency block ensures Task 4 runs after all upstream tasks settle, regardless of their individual outcomes.
5. EIn the DAB job resource YAML, configure Task 4's dependency block to reference Tasks 1, 2, and 3. For each dependency where Task 4 should run regardless of outcome, set the run_if condition at the task level (e.g., run_if: ALL_DONE) so that Task 4 executes after all its upstream dependencies have reached a terminal state (success, failure, or skipped), while not requiring all of them to succeed.