Home / DE Associate practice test / Troubleshooting, Monitoring, and Optimization / Question

Databricks Data Engineer Associate · free practice question

A stage runs far slower than expected. In the Spark UI stage details the data engineer sees large non-zero values under the Spill (memory) and Spill (disk)…

Exam-style question on Troubleshooting, Monitoring, and Optimization from the free DE Associate practice test, with the answer and a full explanation below.

Updated November 10, 2026

Troubleshooting, Monitoring, and Optimization

A stage runs far slower than expected. In the Spark UI stage details the data engineer sees large non-zero values under the Spill (memory) and Spill (disk) columns, even though task input sizes look balanced across all tasks. What is happening, and how should it best be addressed?

  1. AExecution memory is exhausted, so tasks serialize partitions to local disk and re-read them, which is far slower; reduce partition size or add memory so partitions fit in execution memory and stop spilling to disk.
  2. BThe stage is broadcasting a table that exceeds the driver's memory, so partitions overflow to disk; lower spark.sql.autoBroadcastJoinThreshold so the large side is no longer broadcast and the spill then disappears.
  3. COne key is skewed and its partition cannot fit in memory; enable AQE skew-join handling, which the spill columns confirm is currently disabled, so Spark can split the oversized partition and avoid the disk writes.
  4. DDelta caching is evicting reused data to disk under memory pressure; call persist(MEMORY_ONLY) on the DataFrame so the cached blocks stay resident in memory and the spill-to-disk metrics in the stage return to zero.
  5. EToo many shuffle partitions are being created, each too small to fill memory; lower spark.sql.shuffle.partitions so fewer, larger partitions are produced and the disk-spill columns in the stage detail drop to zero.
Answer & explanation

Correct answer: A

WHY A: Spill means execution memory ran out, so Spark serializes partitions to local disk and re-reads them (10-100x slower); the fix is to shrink partition size or add memory so partitions fit. WHY NOT B: Spill columns reflect execution memory pressure per task, not driver-side broadcast overflow; balanced inputs argue against a broadcast problem. WHY NOT C: Inputs are balanced across tasks, so this is not skew, and AQE skew handling is on by default rather than disabled. WHY NOT D: Spill (memory)/Spill (disk) metrics track shuffle/aggregation spill, not Delta cache eviction, so persist does not address the cause. WHY NOT E: Fewer, larger partitions would hold more per task and typically increase spill, so lowering shuffle.partitions is the wrong direction here.

Take the full DE Associate practice test free →

Would you pass today?

This question is one of 45 in the free, timed Databricks Data Engineer Associate practice test. Take it and get a section-by-section score.

Start the free test →