Home / DE Professional practice test / Developing Code for Data Processing using Python and SQL / Question

Databricks Data Engineer Professional · free practice question

A data engineering team maintains a Databricks Job that processes data for 12 different retail brands. Each brand has a separate source schema and a slightly…

Exam-style question on Developing Code for Data Processing using Python and SQL from the free DE Professional practice test, with the answer and a full explanation below.

Updated November 16, 2026

Developing Code for Data Processing using Python and SQL Easy

A data engineering team maintains a Databricks Job that processes data for 12 different retail brands. Each brand has a separate source schema and a slightly different transformation logic. Currently, the team maintains 12 nearly identical notebooks, one per brand. A senior engineer wants to refactor this into a single parameterized notebook that uses control flow to select the brand-specific logic at runtime, driven by a job task parameter. Additionally, the team wants the job to dynamically iterate over all 12 brands in parallel using a Databricks Workflow construct rather than sequential task execution. Which combination of Databricks features correctly implements this dynamic parallel execution pattern?

  1. AUse one notebook with an if/elif chain on dbutils.widgets.get('brand'). Create 12 separate jobs, one per brand with its own brand parameter, and wrap them in a parent job whose shell tasks call databricks jobs run-now for the 12 jobs in parallel.
  2. BUse one parameterized notebook with an if/elif block or dictionary dispatch on dbutils.widgets.get('brand'). Configure a for_each task that iterates over the list of brand names and runs the notebook as a nested task per brand, with a configurable concurrency.
  3. CUse one notebook with a Python loop inside a single task: for brand in brand_list: process_brand(brand). This is preferred over the for_each construct, as in-task processing avoids the scheduling overhead of 12 separate tasks and costs less for small workloads.
  4. DUse a Lakeflow Spark Declarative Pipeline with a @dlt.table function that takes a brand parameter. Run it in CONTINUOUS mode with the brand names as a JSON array in the pipeline configuration, and the pipeline fans out one parallel run per brand.
  5. EUse a multi-task job with 12 tasks defined in the job YAML with depends_on: [], each referencing the shared notebook with a different brand value in its task parameters. The static tasks run in parallel and need no iteration construct such as for_each.
Answer & explanation

Correct answer: B

WHY B: This is the canonical Databricks pattern for dynamic parallel fan-out. The for_each task iterates over a dynamic input list (defined at run time or in the job configuration) and invokes a child task (notebook, Python script, etc.) for each item, with configurable concurrency. This is more flexible than static multi-task definitions because the brand list can change without modifying the job definition. The parameterized notebook reads brand via dbutils.widgets.get() and uses Python control flow (if/elif or a dispatch dictionary) to select brand-specific logic.

WHY NOT A: Maintaining 12 separate jobs is the anti-pattern that the refactoring is trying to eliminate. A parent orchestration job using shell tasks adds unnecessary complexity.

WHY NOT C: A for loop within a single task processes brands sequentially, eliminating parallelism. The 12 brands may have large independent datasets, and parallel execution is explicitly required.

WHY NOT D: Lakeflow SDP does not support per-brand fan-out from a single pipeline config. Pipelines are defined as a single DAG; there is no built-in @dlt.table parameter mechanism for running parallel pipeline instances.

WHY NOT E: A static definition with 12 hardcoded tasks is inflexible: adding a 13th brand requires modifying the job definition. The for_each construct handles dynamic lists without job definition changes.

Take the full DE Professional practice test free →

Would you pass today?

This question is one of 60 in the free, timed Databricks Data Engineer Professional practice test. Take it and get a section-by-section score.

Start the free test →