Debugging and Deploying: Databricks Data Engineer Professional Practice Questions

Correct answer: A

WHY A is correct: The pipeline event log is the primary tool for extracting detailed observability data from Lakeflow Spark Declarative Pipeline updates. It records data lineage (which source datasets fed which outputs), data quality metrics (expectation name, number of records that passed vs. failed per constraint, records dropped or quarantined), and resource usage for each pipeline update. When data quality expectations are causing unexpected record counts, the event log is the correct diagnostic resource — it provides per-expectation pass/fail breakdowns that are not available in any other single location. WHY NOT C: The cluster Metrics tab on the Compute page shows hardware metrics (CPU, memory) and Spark job-level metrics (active tasks, shuffle bytes). It does not contain dataset-level data quality metrics such as expectation pass/fail counts or record counts filtered by pipeline expectations. WHY NOT B: system.lakeflow.job_task_run_timeline contains per-task run metadata for Databricks Jobs (start time, end time, result state). Pipelines are not structured as multi-task jobs in the same way, and this table does not contain data quality metrics such as expectation enforcement outcomes. WHY NOT D: The Spark UI's SQL tab shows the execution plans and logical/physical plans for SQL queries, including estimated and actual row counts at the operator level. While useful for understanding query execution, it does not surface pipeline-level data quality metrics tied to specific EXPECT constraint violations or dropped record counts by expectation name.

Correct answer: C

WHY C is correct: A Repair Run in Databricks re-runs only the failed tasks and skipped tasks, along with any of their downstream dependents. In this scenario, Task C failed and Tasks D and E were skipped because they depend on C. A repair run will re-execute Tasks C, D, and E. Tasks A and B, which completed successfully, are NOT re-run — this is the explicit design: the repair run preserves successful task results, saving time and compute cost. This behavior is documented as a key property: 'A repair run only runs failed and skipped tasks. Successfully completed tasks are not re-run.' WHY NOT A: Re-running only Task C without its dependents (D and E) would leave the job in an incomplete state if C succeeds. Databricks repair runs automatically include downstream dependents of the repaired tasks to ensure the job can finish successfully. WHY NOT B: Re-running all 5 tasks defeats the purpose of the repair run feature, which is designed to save time and resources by preserving the results of tasks that already completed successfully. Databricks explicitly does NOT re-run successfully completed tasks during a repair. WHY NOT D: This is the same as B — repair runs do not restart from the first task. They start from the failed/skipped tasks and proceed through dependents only.

Correct answer: A

WHY A is correct: The system.lakeflow.job_task_run_timeline table in the Unity Catalog system catalog records per-task execution details for each job run, including task_name, run_start_time, run_end_time, result_state, and other identifying fields. It is the correct table when troubleshooting which tasks within a multi-task job succeeded, failed, or were skipped — providing the granular, per-task diagnostic information the engineer needs. System tables in the system catalog are Databricks-managed, read-only Delta Sharing tables that require Unity Catalog. Access requires USE CATALOG and SELECT on the relevant tables (account admins have access by default); data is retained for 365 days in most system tables. WHY NOT B: system.billing.usage records cost and usage metrics such as DBU consumption by product, SKU, and workspace. It is used for cost analysis and chargeback, not for identifying which tasks within a job run failed or their execution times. WHY NOT C: system.compute.node_timeline records cluster node-level hardware events such as node additions, removals, and spot evictions. It is useful for diagnosing cluster scaling and node failures, but does not contain per-task job run details such as task start time, end time, or result state. WHY NOT D: system.access.audit records access audit events — who accessed what data, when, and from which workspace. It is used for security auditing and compliance, not for diagnosing job task failures or execution timelines.

Correct answer: C

WHY C is correct: The Databricks Asset Bundles lifecycle follows a defined sequence: databricks bundle init (scaffold a new bundle from a template) → edit databricks.yml → databricks bundle validate (check config for errors) → databricks bundle deploy (create or update resources in the workspace) → databricks bundle run (execute a specific job or pipeline). After a successful validate, the correct next step to push the defined resources into the workspace is databricks bundle deploy --target <target_name>. This command creates the jobs, pipelines, or other defined resources in the specified target environment. The Databricks CLI v0.218.0+ is required. WHY NOT A: databricks bundle init is used to scaffold a brand-new bundle project from a template into a local directory. It is a one-time initialization step at the start of creating a bundle and is not used after the bundle is already defined and validated. WHY NOT B: databricks bundle run executes a specific resource (such as a job or pipeline) that has already been deployed to the workspace. Running a bundle before deploying it would fail because the referenced resources do not yet exist in the target workspace. The correct sequence is deploy first, then run. WHY NOT D: databricks bundle publish is not a standard Databricks Asset Bundles CLI command for deploying workspace resources. The correct command is databricks bundle deploy. There is a bundle publish concept for sharing bundles as templates, but it is not the deployment command for jobs and pipelines.

Correct answer: A

WHY A is correct: Databricks documentation describes the correct pattern for production Git folders as follows: (1) They must be created by admins OUTSIDE user folders — under /Workspace/ rather than /Workspace/Users/, often organized by team or project. (2) They hold the deployment branch (e.g., main) and serve as the source for automated workflows. (3) Access should be restricted — most users receive only 'Can Run' to execute workflows against the folder, while admins and service principals used for automation receive edit access. (4) Synchronization should be automated via external CI/CD tools (e.g., GitHub Actions) or a scheduled Databricks job that calls w.repos.update() via the Workspace SDK when PRs are merged into the deployment branch. This prevents manual drift and unauthorized direct edits. WHY NOT B: Placing the production Git folder under a user directory (/Workspace/Users/<admin>/) is explicitly incorrect — user directories are for individual developer checkouts where they work on branches and push changes. Production folders must be outside user directories so they are not tied to any single user account. Granting Can Edit to all developers also violates the requirement that individual devs cannot directly modify the production folder. WHY NOT C: While placing the folder under user folders is wrong (same issue as A), manually syncing from an admin laptop introduces operational risk: it requires human intervention for every deployment, is error-prone, and does not provide the automated CI/CD workflow the organization requires. Databricks explicitly recommends automating production folder updates. WHY NOT D: Databricks Git Folders do not have a native 'Auto-sync' option that automatically pulls from the remote branch when GitHub detects a push. Automation must be explicitly configured via external CI/CD pipelines (e.g., GitHub Actions webhook-triggered workflow) or scheduled Databricks jobs using the Repos API/Workspace SDK.

Correct answer: D

WHY D is correct: Databricks Asset Bundles explicitly support the following resource types that can be defined, validated, and deployed via databricks.yml: Jobs, Lakeflow Spark Declarative Pipelines (DLT/LDP), AI/BI dashboards, model serving endpoints, MLflow experiments, and MLflow registered models. This makes DABs a comprehensive IaC approach for the entire Databricks ML platform — from feature pipelines and model training to model registry and serving infrastructure — all managed from version-controlled YAML files using the Databricks CLI v0.218.0+. WHY NOT A: This understates DAB capabilities. Databricks Asset Bundles explicitly list model serving endpoints, MLflow experiments, and MLflow registered models as supported resource types. Limiting management of these assets to the MLflow API or UI defeats the IaC benefits DABs provide. WHY NOT C: MLflow experiments and registered models ARE supported resource types in Databricks Asset Bundles. They can be declared as resources in databricks.yml and deployed via databricks bundle deploy. This option incorrectly excludes them. WHY NOT B: Databricks Asset Bundles are an infrastructure-as-code tool for defining and deploying Databricks workspace resources. They are client-side CLI tooling and do not impose Databricks Runtime version constraints on job or pipeline clusters. Runtime version requirements depend on the features used within the jobs and pipelines themselves, not on the DAB toolchain.

Correct answer: B

WHY B is correct: The Databricks documentation explicitly states that the pipeline Monitor UI allows users to 'View metrics for streaming sources, like Apache Kafka and Auto Loader.' The Monitor UI shows real-time progress and status of pipeline updates including streaming-specific metrics such as input rate, processing rate, batch duration, and trigger interval. This is the designed, purpose-built surface for real-time monitoring of an active pipeline update and is the first place to look when diagnosing streaming pipeline lag. WHY NOT A: The pipeline event log stores detailed update information and is the right tool for post-run analysis and data quality auditing. While it does record flow progress events that contain batch statistics, it is not a real-time streaming dashboard — it is a historical log that is queried after or during an update, not a live metrics display. The Monitor UI is the real-time observation surface. WHY NOT C: system.lakeflow.pipeline_update_timeline records pipeline update-level metadata (update start time, end time, result state, triggered by). It tracks update lifecycle events, not intra-update streaming metrics like input rate per micro-batch or batch processing duration. Additionally, using Structured Streaming to query a Delta Sharing system table adds latency and complexity that is unnecessary when the Monitor UI is available. WHY NOT D: The cluster Metrics tab shows Spark and hardware metrics for the cluster underlying the pipeline but does not expose streaming source-specific metrics like Kafka consumer lag, per-batch input row counts, or trigger intervals. Kafka consumer group offsets are tracked by the Kafka broker, not surfaced in the Databricks cluster Metrics tab.

Correct answer: B

WHY B is correct: The Metrics tab on the Compute page (cluster detail page) in the Databricks UI displays two categories of cluster metrics: hardware metrics (CPU utilization, memory usage, network I/O, filesystem I/O) and Spark metrics (active tasks, failed tasks, shuffle read bytes, shuffle write bytes, task duration distribution). Metrics are collected approximately every minute with less than 1 minute of delay, and the history is retained for 30 days. This is the designed UI surface for diagnosing cluster-level performance bottlenecks on all-purpose and jobs compute clusters. WHY NOT A: system.lakeflow.job_run_timeline records job run-level metadata such as run start time, end time, result state, and triggering user. It does not contain hardware or Spark performance metrics like CPU utilization or shuffle bytes; those are surfaced in the Compute Metrics UI, not in system tables. WHY NOT C: The Spark UI (accessible from the cluster or job run) provides Spark DAG-level information such as stage durations, task counts, and shuffle volumes per stage. However, it does not surface hardware-level metrics like CPU utilization or memory for the cluster nodes at the 'Stages' tab level. The Metrics tab on the Compute page is the correct location for those hardware metrics. WHY NOT D: The cluster Event Log tab records cluster lifecycle events such as when the cluster was started, configured, resized (autoscaling), or terminated. It does not contain time-series performance metrics like CPU utilization, memory usage, or Spark task metrics.