A task occasionally fails due to a transient network timeout when calling an external API, but succeeds when re-run. Without adding external tooling, what is the appropriate Lakeflow Jobs configuration to make the pipeline resilient to these intermittent failures?
Answer & explanation
Correct answer: E
WHY E: A task retry policy re-runs the task automatically up to a configured limit with a set interval. It is the built-in mechanism for transient, self-resolving failures.
WHY NOT B: ALL_DONE is a run-if condition that controls whether a task runs after its upstream tasks finish; it does not retry the failing task, and masking a failure as success is wrong.
WHY NOT C: Running ten copies is wasteful, repeats the external API call ten times, and is not how job resilience is configured.
WHY NOT D: Checkpoints aid recovery; disabling them harms resilience rather than helping it, and nothing here re-runs the task within the same job run.
WHY NOT A: A larger cluster does not prevent an external service's network timeouts.