Home / DE Professional practice test / Developing Code for Data Processing using Python and SQL / Question

Databricks Data Engineer Professional · free practice question

A data engineering team is building a production Databricks Job that uses a custom internal Python library (`mycompany-utils==2.3.1`) published to a private…

Exam-style question on Developing Code for Data Processing using Python and SQL from the free DE Professional practice test, with the answer and a full explanation below.

Updated November 16, 2026

Developing Code for Data Processing using Python and SQL Medium

A data engineering team is building a production Databricks Job that uses a custom internal Python library (mycompany-utils==2.3.1) published to a private PyPI repository, plus two public PyPI packages: pandas==2.0.3 and scikit-learn==1.4.0. The job runs on a Job Cluster with Databricks Runtime 14.3 LTS. The team has three requirements: (1) the custom library must be version-pinned to prevent accidental upgrades, (2) all dependencies must be available at the start of the first task, (3) the solution must work in a CI/CD pipeline without manual steps. Which cluster library configuration approach correctly satisfies all three requirements?

  1. AAdd an %pip install magic command at the top of the first notebook in the job. This installs the packages at notebook startup, ensures they are available before any transformation code runs, and the exact version pins in the install command satisfy the version-locking requirement. No cluster-level configuration is needed.
  2. BConfigure the Job Cluster's libraries block in the DAB job resource YAML to include all three packages with exact version pins using the pypi source type. This installs the dependencies when the cluster starts, making them available to all tasks before any notebook runs, and the YAML configuration is version-controlled and CI/CD-compatible.
  3. CInstall all three packages as cluster-level libraries on a long-running All-Purpose cluster and configure the job to use that cluster instead of a Job Cluster. Cluster-level libraries persist across job runs, guaranteeing that dependencies are always available. Version pinning is enforced by the cluster's library list, which is managed via the Workspace UI.
  4. DBuild a custom Docker container image that includes all three libraries, publish it to a private container registry, and configure the Job Cluster to use this custom container via the docker_image cluster configuration. This guarantees complete dependency isolation, exact version pinning, and works in CI/CD pipelines, but requires Databricks Runtime for Machine Learning (not LTS) to enable container-based clusters.
  5. EUpload a requirements.txt file listing all three packages with version pins to DBFS, and reference it in the job's cluster configuration using spark.databricks.pip.requirements Spark configuration property. Databricks automatically installs the listed packages during cluster initialization using this property, satisfying all three requirements without any code changes.
Answer & explanation

Correct answer: B

WHY B: Configuring libraries in the DAB job YAML libraries block is the correct, CI/CD-native approach. Libraries declared at the cluster level in a job definition are installed during cluster initialization — before any task runs — ensuring availability from task one. Version pinning is expressed directly in the YAML (e.g., pypi: {package: 'pandas==2.0.3'}), and the YAML is version-controlled in Git. Private PyPI repos are supported via the repo field or workspace-configured secret-backed repo credentials. WHY NOT A: %pip install in a notebook restarts the Python kernel after installation, which can cause issues with shared cluster state and does not guarantee availability before other tasks. For multi-task jobs, each task notebook would need its own install command, introducing duplication and fragility. WHY NOT C: Using an all-purpose cluster violates the job isolation principle, and managing library versions through the UI is not CI/CD-compatible. WHY NOT D: Custom Docker containers require Databricks Runtime for Machine Learning or a custom container policy, which is not guaranteed for standard LTS runtimes. This approach is heavier than necessary and adds container build/publish complexity. WHY NOT E: spark.databricks.pip.requirements referencing a DBFS path is not a standard or officially supported cluster initialization mechanism for pip dependencies. The correct supported methods are cluster library configuration and %pip install.

Take the full DE Professional practice test free →

Would you pass today?

This question is one of 60 in the free, timed Databricks Data Engineer Professional practice test. Take it and get a section-by-section score.

Start the free test →