Home / DE Professional practice test / Developing Code for Data Processing using Python and SQL / Question

Databricks Data Engineer Professional · free practice question

A data engineering team is deploying a Databricks Job with a Lakeflow Spark Declarative Pipeline task. The pipeline runs in `TRIGGERED` mode and is expected to…

Exam-style question on Developing Code for Data Processing using Python and SQL from the free DE Professional practice test, with the answer and a full explanation below.

Updated November 16, 2026

Developing Code for Data Processing using Python and SQL Medium

A data engineering team is deploying a Databricks Job with a Lakeflow Spark Declarative Pipeline task. The pipeline runs in TRIGGERED mode and is expected to fully process all pending data within 30 minutes under normal conditions. The business SLA requires the pipeline to complete within 45 minutes. The team wants the pipeline to automatically stop and alert if it has not completed within 45 minutes, and they want the pipeline cluster to automatically terminate after completion to avoid idle cluster costs. The team is also aware of a known upstream data quality issue that occasionally causes transient failures, and they want a maximum of 2 automatic retries before the alert is raised. Which DAB job configuration correctly implements all three requirements?

  1. ASet timeout_seconds: 2700 (45 minutes) at the pipeline task level to stop the pipeline if it exceeds the SLA. Set max_retries: 2 at the task level to allow up to 2 retries. Pipeline clusters auto-terminate when the job finishes by default — no additional configuration is needed for cluster termination.
  2. BSet timeout_seconds: 2700 at the job level (not task level) so the entire job run is terminated if it exceeds 45 minutes total. Set max_retries: 2 at the job level to allow retries of the entire job. Pipeline clusters auto-terminate when the TRIGGERED pipeline completes. Configure a job-level on_failure notification to alert when the job fails after exhausting retries.
  3. CSet timeout_seconds: 2700 at the task level. Set max_retries: 2 with min_retry_interval_millis: 30000 (30 seconds between retries) to avoid immediately hammering the transient failure source. Configure an on_failure job notification. Pipeline clusters (in TRIGGERED mode) auto-terminate when the pipeline run completes, so no explicit auto-termination config is needed.
  4. DSet timeout_seconds: 1800 (30 minutes, matching the expected runtime) to ensure the pipeline is strictly bounded to its normal execution time. Set max_retries: 2 at the task level. Add autotermination_minutes: 10 to the pipeline cluster configuration to terminate the cluster 10 minutes after the pipeline completes, providing a buffer for any post-processing cleanup tasks.
  5. EConfigure the pipeline in CONTINUOUS mode with a timeout_seconds: 2700 at the task level. CONTINUOUS mode automatically retries internally on transient failures without the job needing to retry the task. Set max_retries: 0 at the task level since CONTINUOUS mode handles its own resilience, and configure the on_failure notification at the job level.
Answer & explanation

Correct answer: C

WHY C: timeout_seconds: 2700 at the task level is the correct SLA enforcement — the pipeline task is terminated if it does not complete within 45 minutes. max_retries: 2 allows up to two automatic retries for transient failures before the job is considered permanently failed. min_retry_interval_millis adds a configurable delay between retries, which is a best practice for transient upstream issues (avoids immediately retrying a still-failing source). Pipeline clusters in TRIGGERED mode auto-terminate when the pipeline run completes — this is a built-in behavior of TRIGGERED mode job clusters, so no explicit termination config is needed. on_failure notification triggers after all retries are exhausted. WHY NOT A: While functionally similar, the task-level timeout without a retry interval is less correct than C because immediate retries on transient issues may repeatedly hit the same failure. Option C is more complete and operationally sound. WHY NOT B: Setting timeout and retries at the job level (rather than task level) is less precise for multi-task jobs, as the timeout would apply to the entire job's wall-clock time including scheduling overhead. WHY NOT D: Setting timeout_seconds: 1800 (the expected runtime) as the timeout means any run that takes slightly longer than normal (but still within SLA) would be killed unnecessarily. The SLA is 45 minutes; the timeout should match the SLA, not the expected runtime. WHY NOT E: CONTINUOUS mode is for ongoing, low-latency streaming. Using it for a 30-minute TRIGGERED batch pipeline is inappropriate and would keep the cluster running indefinitely rather than terminating after each batch.

Take the full DE Professional practice test free →

Would you pass today?

This question is one of 60 in the free, timed Databricks Data Engineer Professional practice test. Take it and get a section-by-section score.

Start the free test →