A data engineering team is deploying a Databricks Job with a Lakeflow Spark Declarative Pipeline task. The pipeline runs in TRIGGERED mode and is expected to fully process all pending data within 30 minutes under normal conditions. The business SLA requires the pipeline to complete within 45 minutes. The team wants the pipeline to automatically stop and alert if it has not completed within 45 minutes, and they want the pipeline cluster to automatically terminate after completion to avoid idle cluster costs. The team is also aware of a known upstream data quality issue that occasionally causes transient failures, and they want a maximum of 2 automatic retries before the alert is raised. Which DAB job configuration correctly implements all three requirements?
Answer & explanation
Correct answer: C
WHY C: timeout_seconds: 2700 at the task level is the correct SLA enforcement — the pipeline task is terminated if it does not complete within 45 minutes. max_retries: 2 allows up to two automatic retries for transient failures before the job is considered permanently failed. min_retry_interval_millis adds a configurable delay between retries, which is a best practice for transient upstream issues (avoids immediately retrying a still-failing source). Pipeline clusters in TRIGGERED mode auto-terminate when the pipeline run completes — this is a built-in behavior of TRIGGERED mode job clusters, so no explicit termination config is needed. on_failure notification triggers after all retries are exhausted. WHY NOT A: While functionally similar, the task-level timeout without a retry interval is less correct than C because immediate retries on transient issues may repeatedly hit the same failure. Option C is more complete and operationally sound. WHY NOT B: Setting timeout and retries at the job level (rather than task level) is less precise for multi-task jobs, as the timeout would apply to the entire job's wall-clock time including scheduling overhead. WHY NOT D: Setting timeout_seconds: 1800 (the expected runtime) as the timeout means any run that takes slightly longer than normal (but still within SLA) would be killed unnecessarily. The SLA is 45 minutes; the timeout should match the SLA, not the expected runtime. WHY NOT E: CONTINUOUS mode is for ongoing, low-latency streaming. Using it for a 30-minute TRIGGERED batch pipeline is inappropriate and would keep the cluster running indefinitely rather than terminating after each batch.