A data engineering team has a Lakeflow Spark Declarative Pipeline that ingests from 5 different Auto Loader sources into 5 separate Bronze Streaming Tables, feeds into 3 Silver Streaming Tables (with joins and quality constraints), and produces 2 Gold Materialized Views. The pipeline runs in TRIGGERED mode every hour. During a recent incident, the team discovered that a failure in one of the 5 Auto Loader sources caused the entire pipeline to fail, blocking all Silver and Gold updates even for the unaffected sources. The team wants to improve pipeline resilience so that failures in one Bronze source do not block other independent pipeline branches. Which architectural change BEST addresses this requirement?
Answer & explanation
Correct answer: B
WHY B: In a single Lakeflow SDP pipeline, the tables share the same pipeline execution context and checkpoint, and a failure in any table definition can halt the entire pipeline update. Splitting into independent pipelines provides true fault isolation: each pipeline has its own checkpoint, its own error boundary, and its own retry/failure state. Job orchestration (Lakeflow Jobs, formerly Databricks Workflows) lets the independent pipelines run in parallel with configurable concurrency, and a failure in one pipeline's run does not affect the others. This is the recommended architectural pattern for resilient multi-source pipelines.
WHY NOT A: delta.schema.autoMerge relates to schema evolution during Delta Lake write operations, not to source-level Auto Loader failures. It does not prevent cascading failures from a failing source.
WHY NOT C: cloudFiles.allowOverwrites controls whether Auto Loader reprocesses files that were overwritten after being processed. It is not related to error isolation between multiple sources.
WHY NOT D: Warn-level expectations handle data quality constraint violations (for example null values or type mismatches in data). A completely failing Auto Loader source (permission error, connectivity issue) is not a constraint violation; it is an infrastructure/source error that halts the streaming read, not a row-level data quality event.
WHY NOT E: pipelines.trigger.interval controls how frequently a flow is triggered. Shorter intervals do not change the error isolation model: the sources still share the same pipeline execution context, and a failure in any source still fails the pipeline run.