A data engineer is troubleshooting a long-running Databricks job on an all-purpose cluster. They want to inspect CPU utilization, memory usage, and Spark-level metrics such as active tasks, failed tasks, and shuffle read/write bytes over the past several hours to identify the source of the slowness. Which approach is CORRECT?
Show answer & explanation
Correct answer: A
WHY A: The Metrics tab on the Compute page (cluster detail page) in the Databricks UI displays two categories of cluster metrics: hardware metrics (CPU utilization, memory usage, network I/O, filesystem I/O) and Spark metrics (active tasks, failed tasks, shuffle read bytes, shuffle write bytes, task duration distribution). Metrics are collected approximately every minute with less than 1 minute of delay, and the history is retained for 30 days. This is the designed UI surface for diagnosing cluster-level performance bottlenecks on all-purpose and jobs compute clusters.
WHY NOT C: system.lakeflow.job_run_timeline records job run-level metadata such as run start time, end time, result state, and triggering user. It does not contain hardware or Spark performance metrics like CPU utilization or shuffle bytes; those are surfaced in the Compute Metrics UI.
WHY NOT B: The Spark UI provides Spark DAG-level information such as stage durations, task counts, and shuffle volumes per stage. Its Stages tab does not surface hardware-level metrics like CPU utilization or memory for the cluster nodes; the Metrics tab on the Compute page is the location for those.
WHY NOT D: The cluster Event Log tab records cluster lifecycle events such as when the cluster was started, configured, resized (autoscaling), or terminated. It does not contain time-series performance metrics like CPU utilization, memory usage, or Spark task metrics.