Home / DE Professional practice test / Debugging and Deploying

Free · 8 questions with explanations

Debugging and Deploying: Databricks Data Engineer Professional Practice Questions

Exam-style questions on Debugging and Deploying. Pick your answer, then open the explanation to see why it's right — and why the other options are wrong.

Updated November 16, 2026

1 Debugging and Deploying Medium

A data engineer is troubleshooting a long-running Databricks job on an all-purpose cluster. They want to inspect CPU utilization, memory usage, and Spark-level metrics such as active tasks, failed tasks, and shuffle read/write bytes over the past several hours to identify the source of the slowness. Which approach is CORRECT?

  1. AOpen the Compute page, select the cluster, and view the Metrics tab for the hardware and Spark charts over time.
  2. BOpen the Spark UI for the active job run and review the Stages tab for CPU and memory usage broken down by cluster node.
  3. CQuery system.lakeflow.job_run_timeline filtered by cluster_id to retrieve CPU utilization and shuffle read/write metrics.
  4. DOpen the cluster's Event Log tab, which records CPU and memory utilization alongside autoscaling and failed-task events.
Show answer & explanation

Correct answer: A

WHY A: The Metrics tab on the Compute page (cluster detail page) in the Databricks UI displays two categories of cluster metrics: hardware metrics (CPU utilization, memory usage, network I/O, filesystem I/O) and Spark metrics (active tasks, failed tasks, shuffle read bytes, shuffle write bytes, task duration distribution). Metrics are collected approximately every minute with less than 1 minute of delay, and the history is retained for 30 days. This is the designed UI surface for diagnosing cluster-level performance bottlenecks on all-purpose and jobs compute clusters. WHY NOT C: system.lakeflow.job_run_timeline records job run-level metadata such as run start time, end time, result state, and triggering user. It does not contain hardware or Spark performance metrics like CPU utilization or shuffle bytes; those are surfaced in the Compute Metrics UI. WHY NOT B: The Spark UI provides Spark DAG-level information such as stage durations, task counts, and shuffle volumes per stage. Its Stages tab does not surface hardware-level metrics like CPU utilization or memory for the cluster nodes; the Metrics tab on the Compute page is the location for those. WHY NOT D: The cluster Event Log tab records cluster lifecycle events such as when the cluster was started, configured, resized (autoscaling), or terminated. It does not contain time-series performance metrics like CPU utilization, memory usage, or Spark task metrics.

2 Debugging and Deploying Easy

A Databricks job run fails because a job parameter --env was mistakenly set to staging when the code requires production to connect to the correct data source. The engineer needs to re-run only the failed tasks using --env=production, WITHOUT permanently modifying the original job definition. What is the CORRECT approach?

  1. AEdit the job definition to change the default value of --env to production, then trigger a new full run of the job.
  2. BClone the job, set --env to production in the cloned copy, run the clone once, and then delete the clone afterwards.
  3. CIn the Repair job run dialog, override --env to production for that repair run, leaving the job definition unchanged.
  4. DCancel the failed run, then submit a new run through the Jobs REST API run-now call with --env set to production.
Show answer & explanation

Correct answer: C

WHY C: The Repair job run dialog in the Databricks UI supports parameter overrides for the repair run. When initiating a repair, engineers can add or change task parameters and job-level parameters that apply only to the repair execution; the original job definition remains unchanged. This is the designed workflow for remediating a failed run caused by a bad parameter value while preserving the job's original configuration and the results of tasks that already completed successfully. WHY NOT A: Editing the job definition changes the permanent default parameter, which would affect future runs. It also requires triggering a completely new run from scratch, re-executing tasks that may have already completed successfully, which wastes compute and loses the efficiency benefit of the repair run mechanism. WHY NOT B: Cloning the job is an unnecessarily complex workaround. It requires cleanup after the fact, does not use the built-in repair run functionality, and triggers a full new run that re-executes every task (including tasks that already succeeded) rather than just the failed ones. WHY NOT D: Submitting a new run via the API with the corrected parameter is a valid workaround but creates an entirely new run rather than repairing the original. This wastes compute by re-running successful tasks and creates an additional run record, complicating run history tracking. It also does not preserve the original run's context for audit purposes.

3 Debugging and Deploying Medium

An organization wants to set up a production Git folder in Databricks that tracks the main branch of a remote GitHub repository. The folder must be automatically updated when PRs are merged into main, and individual developers must NOT be able to directly edit files in it. Which setup CORRECTLY achieves this?

  1. ACreate the Git folder under /Workspace/Users/<admin>/ and grant Can Edit to the developers, who commit hotfixes directly in the production folder after each merged PR.
  2. BCreate the Git folder under /Workspace/ outside user folders, limit Can Edit to admins and a service principal, and pull main with GitHub Actions or a scheduled job.
  3. CCreate the Git folder under /Workspace/Users/<admin>/, grant Can Run to the users, and sync it by hand each deployment cycle with the Repos API from an admin laptop.
  4. DCreate the Git folder anywhere in the workspace and enable its Auto-sync option, which makes Databricks pull from the remote branch whenever GitHub detects a push.
Show answer & explanation

Correct answer: B

WHY B: Databricks documentation describes the pattern for production Git folders: (1) they are created by admins OUTSIDE user folders, under /Workspace/ rather than /Workspace/Users/, often organized by team or project; (2) they hold the deployment branch (e.g., main) and serve as the source for automated workflows; (3) access is restricted: most users, and the deployment service principal for running, receive 'Can Run' to execute workflows against the folder, while admins and the service principals used for automation receive edit access; (4) synchronization is automated via external CI/CD tools (e.g., GitHub Actions) or a scheduled Databricks job that calls w.repos.update() via the Workspace SDK when PRs are merged into the deployment branch. This prevents manual drift and unauthorized direct edits. WHY NOT A: A production Git folder under a user directory (/Workspace/Users/<admin>/) is incorrect: user directories are for individual developer checkouts where they work on branches and push changes, and a production folder should not be tied to a single user account. Granting Can Edit to the developers also violates the requirement that individual developers do not edit the production folder directly. WHY NOT C: The folder is again under a user directory (same issue as A), and syncing by hand from an admin laptop needs human intervention for each deployment, is error-prone, and is not the automated update on merge that the organization requires. Databricks recommends automating production folder updates. WHY NOT D: Databricks Git folders have no native 'Auto-sync' option that pulls from the remote branch when GitHub detects a push. Automation is configured explicitly via external CI/CD pipelines (e.g., a GitHub Actions workflow triggered on merge) or scheduled Databricks jobs using the Repos API/Workspace SDK.

4 Debugging and Deploying Easy

A data engineering team has defined a Databricks job and a Lakeflow pipeline in a databricks.yml bundle configuration file. They have already run databricks bundle validate without errors. What is the CORRECT next command to create or update these resources in a target environment named production?

  1. Adatabricks bundle deploy --target production
  2. Bdatabricks bundle run --target production
  3. Cdatabricks bundle init --target production
  4. Ddatabricks bundle publish --target production
Show answer & explanation

Correct answer: A

WHY A is correct: The Databricks Asset Bundles lifecycle follows a defined sequence: databricks bundle init (scaffold a new bundle from a template) → edit databricks.yml → databricks bundle validate (check config for errors) → databricks bundle deploy (create or update resources in the workspace) → databricks bundle run (execute a specific job or pipeline). After a successful validate, the correct next step to push the defined resources into the workspace is databricks bundle deploy --target <target_name>. This command creates the jobs, pipelines, or other defined resources in the specified target environment. The Databricks CLI v0.218.0+ is required. WHY NOT C: databricks bundle init is used to scaffold a brand-new bundle project from a template into a local directory. It is a one-time initialization step at the start of creating a bundle and is not used after the bundle is already defined and validated. WHY NOT B: databricks bundle run executes a specific resource (such as a job or pipeline) that has already been deployed to the workspace. Running a bundle before deploying it would fail because the referenced resources do not yet exist in the target workspace. The correct sequence is deploy first, then run. WHY NOT D: databricks bundle publish is not a standard Databricks Asset Bundles CLI command for deploying workspace resources. The correct command is databricks bundle deploy. There is a bundle publish concept for sharing bundles as templates, but it is not the deployment command for jobs and pipelines.

5 Debugging and Deploying Easy

A Databricks multi-task job has 5 tasks in sequence: Task A → Task B → Task C → Task D → Task E. During a run, Tasks A and B completed successfully, Task C failed, and Tasks D and E were therefore skipped. The engineer fixes the bug in Task C's notebook and triggers a Repair Run from the Databricks UI. Which tasks will be executed during the repair run?

  1. ATask C will be re-run by itself; Tasks D and E will remain in their original skipped state.
  2. BTasks A through E will re-run from the beginning to give a clean, reproducible execution.
  3. CTasks B, C, D, and E will re-run, as a repair starts one task upstream of the failed task.
  4. DTasks C, D, and E will be re-run; Tasks A and B will keep the results of their earlier runs.
Show answer & explanation

Correct answer: D

WHY D: A Repair Run re-runs the failed tasks and skipped tasks, along with their downstream dependents. Here Task C failed and Tasks D and E were skipped because they depend on C, so the repair run re-executes C, D, and E. Tasks A and B, which completed successfully, are NOT re-run: the repair run preserves successful task results, saving time and compute cost. The documented property is: 'A repair run only runs failed and skipped tasks. Successfully completed tasks are not re-run.' WHY NOT A: Re-running Task C without its dependents would leave the job incomplete even if C succeeds. Repair runs include the skipped downstream dependents of the repaired task so the job can finish. WHY NOT B: Re-running the whole chain defeats the purpose of the repair feature, which saves time and resources by preserving the results of tasks that already completed successfully. Databricks does not re-run successful tasks during a repair. WHY NOT C: A repair does not step back to a successful upstream task. It starts from the failed/skipped tasks and proceeds through their dependents, so Task B is not re-run.

6 Debugging and Deploying Easy

A Databricks workspace team is choosing a CI/CD strategy. Option 1 is to use a production Git folder (updated via the Repos API on PR merge) to deploy notebook code into the workspace. Option 2 is to use Databricks Asset Bundles to define jobs, pipelines, and infrastructure in version-controlled YAML files and deploy them via the CLI. What does Databricks RECOMMEND as the primary CI/CD approach?

  1. AOption 2 (Databricks Asset Bundles) to package and deploy, with Git folders as a complement for source code in the workspace.
  2. BOption 1 (production Git folders), as Git folders natively version-control workspace resources without extra YAML.
  3. CNeither; the Jobs REST API called directly from a CI/CD tool such as GitHub Actions to create and update job definitions.
  4. DOption 1 for notebooks and Option 2 for non-notebook resources, managed independently with two separate toolchains.
Show answer & explanation

Correct answer: A

WHY A: Databricks documentation explicitly states: 'Databricks recommends Databricks Asset Bundles to package and deploy CI/CD workflows.' Bundles provide a full infrastructure-as-code approach covering jobs, pipelines, ML endpoints, and MLflow resources defined in YAML and deployed via the CLI. The documentation also notes that bundles and Git folders are complementary: you can 'Define resources such as jobs and pipelines in source using bundles, then create, deploy, and manage them in workspace Git folders'. Production Git folders remain a valid option when deploying only source code into the workspace, but bundles are the primary recommended strategy. WHY NOT B: Production Git folders are a supported pattern for code-only deployments, but the documentation describes them as the alternative ('If you prefer to deploy only code into the workspace, you can use a production Git folder instead'), not the recommended primary approach. Git folders track code files (notebooks, Python scripts) but do not natively manage resource definitions such as job schedules, cluster configurations, or pipeline settings as first-class versioned objects the way bundles do. WHY NOT C: Calling the Jobs REST API directly is a valid low-level approach, but it requires custom scripting for every resource type and lifecycle operation (create, update, delete, run). Databricks does not recommend it as the primary CI/CD mechanism; it recommends the higher-level abstraction of bundles, which wraps the REST API in a structured, validatable, version-controlled format. WHY NOT D: Databricks does not recommend or document a split toolchain of Git folders for notebooks and bundles for everything else. The documented recommendation is bundles as the unified CI/CD tool for all resource types, with Git folders as a complementary code hosting mechanism within the workspace if desired.

7 Debugging and Deploying Easy

A data engineer needs to investigate why a Lakeflow Spark Declarative Pipeline update processed far fewer records than expected. They suspect that EXPECT data quality constraints are silently dropping or quarantining records. Which tool provides data quality metrics such as expectation pass counts, fail counts, and dropped record counts for each pipeline update?

  1. AThe cluster Metrics tab for the pipeline compute, which shows Spark task shuffle and row counts
  2. BThe system.lakeflow.job_task_run_timeline system table, filtered by the pipeline's pipeline_id
  3. CThe Spark UI's SQL tab, which lists query execution plans and row counts for Delta tables
  4. DThe pipeline event log, which stores data quality metrics and lineage for each pipeline update
Show answer & explanation

Correct answer: D

WHY D: The pipeline event log is the primary tool for detailed observability of Lakeflow Spark Declarative Pipeline updates. It records data lineage (which source datasets fed which outputs), data quality metrics (expectation name, number of records that passed vs. failed per constraint, records dropped), and resource usage for each pipeline update. When expectations are causing unexpected record counts, the event log gives the per-expectation pass/fail breakdown that is not available in another single location. WHY NOT A: The cluster Metrics tab on the Compute page shows hardware metrics (CPU, memory) and Spark job-level metrics (active tasks, shuffle bytes). It does not contain dataset-level data quality metrics such as expectation pass/fail counts. WHY NOT B: system.lakeflow.job_task_run_timeline contains per-task run metadata for Lakeflow Jobs (start time, end time, result state). Pipelines are not structured as multi-task jobs in the same way, and the table holds no expectation enforcement outcomes. WHY NOT C: The Spark UI's SQL tab shows logical/physical execution plans for SQL queries with estimated and actual row counts at the operator level. That helps with query execution, but it does not surface pipeline-level data quality metrics tied to specific EXPECT constraints or dropped record counts by expectation name.

8 Debugging and Deploying Medium

A Lakeflow Spark Declarative Pipeline has a streaming flow ingesting from Apache Kafka but is falling progressively further behind, with the processing lag increasing each batch. The engineer wants to monitor streaming metrics such as input rate, processing rate, batch duration, and trigger interval FOR THE ACTIVE pipeline update in real time. Which tool surface should they use?

  1. AThe pipeline event log, queried with a filter for event_type = 'flow_progress' to get the latest batch statistics
  2. BThe system.lakeflow.pipeline_update_timeline system table, streamed in near-real-time using a Structured Streaming query
  3. CThe Monitor UI for the pipeline, which displays streaming source metrics including Kafka input rate, processing rate, and batch duration
  4. DThe cluster Metrics tab for the pipeline cluster, which shows active Spark tasks and Kafka consumer group offsets
Show answer & explanation

Correct answer: C

WHY C is correct: The Databricks documentation explicitly states that the pipeline Monitor UI allows users to 'View metrics for streaming sources, like Apache Kafka and Auto Loader.' The Monitor UI shows real-time progress and status of pipeline updates including streaming-specific metrics such as input rate, processing rate, batch duration, and trigger interval. This is the designed, purpose-built surface for real-time monitoring of an active pipeline update and is the first place to look when diagnosing streaming pipeline lag. WHY NOT A: The pipeline event log stores detailed update information and is the right tool for post-run analysis and data quality auditing. While it does record flow progress events that contain batch statistics, it is not a real-time streaming dashboard — it is a historical log that is queried after or during an update, not a live metrics display. The Monitor UI is the real-time observation surface. WHY NOT B: system.lakeflow.pipeline_update_timeline records pipeline update-level metadata (update start time, end time, result state, triggered by). It tracks update lifecycle events, not intra-update streaming metrics like input rate per micro-batch or batch processing duration. Additionally, using Structured Streaming to query a Delta Sharing system table adds latency and complexity that is unnecessary when the Monitor UI is available. WHY NOT D: The cluster Metrics tab shows Spark and hardware metrics for the cluster underlying the pipeline but does not expose streaming source-specific metrics like Kafka consumer lag, per-batch input row counts, or trigger intervals. Kafka consumer group offsets are tracked by the Kafka broker, not surfaced in the Databricks cluster Metrics tab.

Take the full DE Professional practice test →