A data engineering team maintains a Databricks Job that processes data for 12 different retail brands. Each brand has a separate source schema and a slightly different transformation logic. Currently, the team maintains 12 nearly identical notebooks, one per brand. A senior engineer wants to refactor this into a single parameterized notebook that uses control flow to select the brand-specific logic at runtime, driven by a job task parameter. Additionally, the team wants the job to dynamically iterate over all 12 brands in parallel using a Databricks Workflow construct rather than sequential task execution. Which combination of Databricks features correctly implements this dynamic parallel execution pattern?
Answer & explanation
Correct answer: B
WHY B: This is the canonical Databricks pattern for dynamic parallel fan-out. The for_each task iterates over a dynamic input list (defined at run time or in the job configuration) and invokes a child task (notebook, Python script, etc.) for each item, with configurable concurrency. This is more flexible than static multi-task definitions because the brand list can change without modifying the job definition. The parameterized notebook reads brand via dbutils.widgets.get() and uses Python control flow (if/elif or a dispatch dictionary) to select brand-specific logic.
WHY NOT A: Maintaining 12 separate jobs is the anti-pattern that the refactoring is trying to eliminate. A parent orchestration job using shell tasks adds unnecessary complexity.
WHY NOT C: A for loop within a single task processes brands sequentially, eliminating parallelism. The 12 brands may have large independent datasets, and parallel execution is explicitly required.
WHY NOT D: Lakeflow SDP does not support per-brand fan-out from a single pipeline config. Pipelines are defined as a single DAG; there is no built-in @dlt.table parameter mechanism for running parallel pipeline instances.
WHY NOT E: A static definition with 12 hardcoded tasks is inflexible: adding a 13th brand requires modifying the job definition. The for_each construct handles dynamic lists without job definition changes.