A data scientist wants to explore a large Databricks Spark DataFrame to understand numeric distributions, string column frequencies, and see inline visual histograms of value distributions — all in a single command without writing extra plotting code. Which tool best satisfies all of these requirements inside a Databricks notebook?
Answer & explanation
Correct answer: D
WHY D: dbutils.data.summarize(df) is the Databricks-specific tool designed to render a full interactive data profile — including summary statistics, null counts, distinct value distributions, and visual histograms for numeric, string, and date columns — in a single call within a Databricks notebook. WHY NOT A: df.summary() returns a statistical summary as a Spark DataFrame but does not automatically render histograms; additional plotting code is required. WHY NOT B: df.describe() provides basic statistics (count, mean, stddev, min, max) but does not render histograms. WHY NOT C: df.stat.freqItems() computes only the most frequent items for specified columns and does not produce a full visual data profile. WHY NOT E: While .toPandas().describe() provides numeric statistics, it does not render histograms automatically and risks out-of-memory errors on large Spark DataFrames; it cannot replicate the inline visual profile produced by dbutils.data.summarize().