A data scientist is building a PySpark MLlib preprocessing Pipeline for a dataset with three columns: annual_salary (double), years_experience (integer), and job_title (string). They plan to use PySpark's Imputer class to handle missing values across all three columns in one step. What is an important limitation they must account for?
Answer & explanation
Correct answer: C
WHY C: The PySpark Imputer documentation explicitly states: 'The imputer does not support categorical features and possibly creates incorrect values for a categorical feature.' The job_title string column cannot be passed to the Imputer; it must be handled separately (for example by filling with the most frequent title using a custom approach or SQL).
WHY NOT A: PySpark's Imputer supports 'mean', 'median', and 'mode' strategies; all three are documented and implemented, so no manual computation is needed.
WHY NOT B: By default the Imputer treats both NaN and null/None as missing for float/double columns (the missingValue parameter defaults to NaN); it does not skip NaN values.
WHY NOT D: The Imputer is a standard MLlib Estimator and is fully compatible with Pipeline stages; it is commonly chained with StringIndexer, OneHotEncoder, and VectorAssembler.
WHY NOT E: The Imputer preserves the original data type of its input columns in the output; it does not cast between double and integer during imputation.