A data scientist wants to clean a salary column in a large Spark DataFrame by removing rows where salary is more than 3 standard deviations above or below the mean. Which PySpark code correctly implements this 3-sigma outlier removal?
Answer & explanation
Correct answer: A
WHY A: This is the correct and efficient 3-sigma implementation. It computes the global mean and standard deviation with a single aggregation, collects the scalar bounds, and applies them as a simple column filter — following the standard two-step pattern (aggregate then filter) recommended for outlier removal in PySpark. WHY NOT B: This approach calls .collect() twice on df_stats, triggering two separate Spark jobs and risking data inconsistency; the correct approach collects once and reuses the scalar values. WHY NOT C: The 0.13th and 99.87th percentiles only approximate ±3 sigma on a perfectly normal distribution; this is not equivalent for skewed distributions and is also not the direct standard deviation method as required. WHY NOT D: StandardScaler with withMean=True and withStd=True does produce z-scores, but this approach requires costly RDD-to-DataFrame conversion and vector wrapping, making it unnecessarily complex for a simple column-level outlier filter. WHY NOT E: Using Window.orderBy() creates a running/cumulative window, not a global aggregation; avg() and stddev() over this window compute running statistics row by row, not the global mean and stddev needed for the 3-sigma rule.