A senior data engineer is designing a new production Python project to be deployed via Databricks Asset Bundles across three environments: dev, staging, and prod. The team consists of 8 engineers who each need to run isolated unit tests locally before pushing code. The project includes shared utility functions used by multiple pipeline notebooks, a set of Lakeflow Spark Declarative Pipeline definitions, and environment-specific configuration values. The engineer wants to ensure the structure is modular, testable, and DAB-compatible. Which project structure and DAB design pattern best satisfies all of these requirements?
Answer & explanation
Correct answer: D
WHY D: This is the canonical scalable DAB project structure. A src/ installable package keeps shared logic modular and locally testable via pip install -e without a live cluster. Separate pipeline definition files keep notebook/pipeline logic decoupled from deployment config. DAB variable substitution (${var.*}) with per-target overrides in databricks.yml is the idiomatic way to handle environment differences without duplicating resource definitions. A tests/ directory with pytest enables fast, cluster-free unit testing.
WHY NOT A: Hardcoding environment-specific values as Python constants makes multi-environment deployment impossible without code changes and violates separation of concerns, and a single main.py is not modular. Inline pipeline SQL inside databricks.yml is not supported.
WHY NOT C: Maintaining three separate databricks.yml files is an anti-pattern that duplicates the resource definitions per environment and introduces drift risk. DAB variable substitution and targets exist precisely to avoid this, and notebook-only logic is hard to unit test locally.
WHY NOT B: Rebuilding a wheel per environment embeds configuration inside an artifact, which is an anti-pattern. Configuration should be external and injected at deploy/runtime, not baked into binary artifacts.
WHY NOT E: DAB does not support Jinja2 templating or a databricks bundle render command. Variable substitution is handled natively by the DAB YAML engine, not via a template rendering step.