A data engineer has built an internal Python library and packaged it as a wheel file (mylib-1.0.0-py3-none-any.whl). The library contains custom data validation logic used by multiple notebooks in a Databricks Job. The engineer needs to make this wheel available to the job cluster. During testing, after uploading the wheel to a Unity Catalog Volume path (/Volumes/dev/libs/wheels/mylib-1.0.0-py3-none-any.whl) and configuring the job cluster to use it, one notebook successfully imports mylib but a second notebook in the same job raises ModuleNotFoundError: No module named 'mylib'. The cluster logs show no installation errors. What is the MOST likely root cause, and what is the correct remediation?
Answer & explanation
Correct answer: A
WHY A: Using %pip install in any notebook on a shared cluster triggers a Python kernel restart for that notebook's process. In some configurations this can conflict with or shadow cluster-level library installations. More critically, in Shared Access Mode clusters, each notebook runs in an isolated Python process — cluster-level libraries are installed into the shared driver/worker environment, but %pip install installs into the per-notebook virtual environment, which takes precedence and can produce inconsistent import behavior across notebooks. The remediation is to ensure a consistent installation method across all notebooks. WHY NOT B: Unity Catalog Volumes (path: /Volumes/...) ARE a supported source for cluster library installation in modern Databricks Runtimes. The wheel path format for Volumes is correctly specified. WHY NOT C: importlib.reload() reloads an already-imported module; it does not fix a ModuleNotFoundError where the module was never found in the first place. WHY NOT D: A malformed wheel would fail installation with an error in the cluster init logs, not install silently. The cluster logs show no installation errors, ruling this out. WHY NOT E: Photon is a vectorized query execution engine for SQL and DataFrame operations. It does not create a separate Python runtime or affect Python library installation paths.