Home / DE Associate practice test / Troubleshooting, Monitoring, and Optimization / Question

Databricks Data Engineer Associate · free practice question

In the Spark UI a data engineer sees one join stage where the max shuffle-read task is 5 GB while the median is 60 MB, and a single straggler task keeps…

Exam-style question on Troubleshooting, Monitoring, and Optimization from the free DE Associate practice test, with the answer and a full explanation below.

Updated November 10, 2026

Troubleshooting, Monitoring, and Optimization

In the Spark UI a data engineer sees one join stage where the max shuffle-read task is 5 GB while the median is 60 MB, and a single straggler task keeps running long after all others finish. AQE is enabled but the skew persists on a few extremely hot keys. Which action best mitigates the skew?

  1. AAdd more executors and larger instances to the cluster so the straggler partition has additional cores available, letting Spark parallelize the single oversized task across the newly added worker nodes in the stage.
  2. BSalt the hot keys by adding a random prefix to spread them across many partitions, join on the salted key, then strip the salt and re-aggregate, which distributes the skewed key's rows far more evenly across tasks.
  3. CSet spark.sql.shuffle.partitions to 1 so the whole join is processed in a single partition, eliminating the per-partition size imbalance that is causing one straggler task to dominate the total stage runtime.
  4. DSet spark.sql.autoBroadcastJoinThreshold to -1 to force Spark to broadcast the larger table to every executor, removing the shuffle on the skewed key so each task completes without waiting on the stragglers.
  5. ECall cache() on the joined DataFrame before the stage runs so the skewed partition is held in memory and reused, preventing the straggler task from recomputing the oversized key on each downstream action.
Answer & explanation

Correct answer: B

WHY B: Salting adds a random prefix to hot keys so their rows spread across many partitions; you join on the salted key then strip and re-aggregate, which is the standard fix when AQE alone cannot tame a few extreme keys. WHY NOT A: More executors do not help because a single oversized partition still runs on one core; the imbalance, not total capacity, is the bottleneck. WHY NOT C: Setting shuffle.partitions to 1 forces everything into one partition, making the straggler problem worse, not better. WHY NOT D: autoBroadcastJoinThreshold = -1 disables auto-broadcast rather than forcing it, and broadcasting a large table would OOM the driver. WHY NOT E: cache() avoids recomputation but does nothing about the uneven partition sizes causing the skew and straggler.

Take the full DE Associate practice test free →

Would you pass today?

This question is one of 45 in the free, timed Databricks Data Engineer Associate practice test. Take it and get a section-by-section score.

Start the free test →