Data Processing: Databricks Machine Learning Associate Practice Questions

A data scientist applies PySpark MLlib's OneHotEncoder to a region column that has 5 distinct category values (indices 0–4 after StringIndexer). The encoder uses the default dropLast=True setting. What is the dimensionality of the resulting sparse vector, and what is the purpose of the dropLast=True default?

1. AThe output vector has 5 elements (one per category), and dropLast=True is set by default to guarantee that every category is represented by its own dedicated binary indicator column, ensuring complete and fully explicit encoding for all 5 categories with no information loss in downstream model training
2. BThe output vector has 4 elements (one fewer than the total number of categories), and dropLast=True is the default behavior to prevent linear dependence among the encoded columns — since the 5 indicators would always sum to exactly 1.0, the last column is mathematically redundant and dropping it avoids the dummy variable trap (perfect multicollinearity) in linear models such as logistic regression and linear SVM
3. CThe output vector has 6 elements (one more than the number of categories), and the additional 6th element serves as a built-in bias term that dropLast=True activates to allow linear models to fit an intercept implicitly through the encoded feature space, eliminating the need for a separate fitIntercept parameter in the downstream model
4. DThe output vector has 4 elements because dropLast=True removes the most frequently occurring category first in order to prevent the model from overfitting to the dominant class — StringIndexer assigns index 0 to the most frequent label, and dropping it by default forces the model to learn from the less common regions which carry more discriminative signal
5. EThe output vector has 5 elements when dropLast=True is active because this default setting adds one extra null-indicator element to the standard 4-element encoding, allowing the model to explicitly represent the case where a row has no valid region value — the extra element is used for missing value representation during inference on new data

A data scientist is preparing features for a linear regression model. The dataset contains a subscription_tier column with 3 nominal categories: 'Bronze', 'Silver', 'Gold'. There is no natural ordering among these tiers for this model's purpose. Which encoding strategy is most appropriate, and why?

1. AUse ordinal integer encoding by assigning Bronze=1, Silver=2, Gold=3 — linear regression requires ordinal numeric inputs and the model will correctly interpret the 1/2/3 values as evenly spaced ordinal levels, learning a single coefficient that scales linearly with tier rank without introducing redundant indicator columns
2. BSkip encoding entirely and pass the raw string column directly to the linear regression model — Spark MLlib's LinearRegression transformer automatically converts string feature values to numeric TF-IDF weighted embeddings before fitting, so manual encoding is unnecessary for nominal categorical columns
3. CUse target encoding (also called mean encoding) because it always outperforms one-hot encoding for linear models on nominal categorical features with fewer than 10 categories — it captures the label signal in a compact single numeric column and eliminates the dimensionality expansion introduced by creating multiple binary indicator columns
4. DUse StringIndexer alone (without a subsequent OneHotEncoder step) — for linear regression, the integer indices 0, 1, and 2 assigned by StringIndexer are sufficiently close in magnitude to binary indicator values that the model can learn separate effects for each tier without the full OHE transformation, which would be redundant given the small number of categories
5. EUse one-hot encoding via StringIndexer followed by OneHotEncoder with dropLast=True, producing 2 binary indicator columns — for a linear regression model, one-hot encoding is appropriate because treating the unordered nominal categories as integers (0, 1, 2) would impose a false linear ordering (Gold > Silver > Bronze) that the model would incorrectly learn as a meaningful numeric gradient in the coefficient

A data scientist wants to understand the relationship between age (continuous, range 18–85) and annual_income (continuous, range $20K–$500K). They want to visualize the relationship and also quantify the strength and direction of any linear association. Which approach is most appropriate?

1. ACreate separate histograms for age and annual_income using different colors on the same plot, and compute the mean and standard deviation of each feature independently — overlapping histograms reveal the degree of linear correlation between two continuous variables by showing where their individual distributions intersect
2. BApply a Chi-Square test of independence after binning both age and annual_income into decile groups — binning both continuous variables and applying the Chi-Square test is the most statistically rigorous method for quantifying linear correlation between two continuous numeric features
3. CCreate a scatter plot with age on the x-axis and annual_income on the y-axis using df.plot.scatter(x='age', y='annual_income'), and compute the Pearson correlation coefficient with df.stat.corr('age', 'annual_income') — the scatter plot reveals the visual pattern and direction of the relationship while the Pearson coefficient quantifies its strength
4. DApply a two-sample t-test comparing the mean annual_income between observations where age is above the median versus below the median — splitting one continuous variable at the median and applying a t-test is a complete and rigorous method for measuring the linear relationship between two continuous numeric features
5. ECreate a box plot of annual_income grouped by quartile bins of age — grouping the income values into four age buckets and displaying box plots for each bucket provides a complete measure of the linear correlation between the two continuous variables without requiring any additional statistical tests

A data analyst wants to determine whether there is a statistically significant association between two nominal categorical variables: customer_segment (values: Premium, Standard, Basic) and preferred_channel (values: Mobile, Web, In-Store). Which approach is most appropriate for comparing these two categorical features?

1. ABuild a contingency table using df.stat.crosstab('customer_segment', 'preferred_channel') to compute the frequency of each segment-channel combination, then apply a Chi-Square test of independence — this is the standard method for testing whether two nominal categorical variables are statistically associated with each other
2. BCompute the Pearson correlation coefficient between numerically label-encoded values of customer_segment and preferred_channel using df.stat.corr() — Pearson correlation is the standard statistical measure for quantifying the strength and direction of the linear relationship between two categorical variables when they are represented as integers
3. CCreate a scatter plot with customer_segment on the x-axis and preferred_channel on the y-axis — scatter plots reveal the joint distribution and strength of association between two nominal categorical variables and are the most commonly used comparison method in exploratory data analysis on string-valued features
4. DCompute the mean of a label-encoded preferred_channel grouped by customer_segment using df.groupBy('customer_segment').agg(F.mean('preferred_channel')) — averaging the label-encoded channel values per segment gives an interpretable numeric measure of which channel each segment prefers most
5. EApply an independent samples t-test comparing the distribution of a numerically encoded preferred_channel across different levels of customer_segment — the two-sample t-test is the appropriate hypothesis test for comparing two nominal categorical distributions when each category group can be treated as an independent numeric sample

A machine learning engineer is performing exploratory data analysis on a house_price column, which is a continuous numeric feature. They want to understand the shape of its distribution — specifically whether it is approximately normal, right-skewed, or multimodal. Which visualization is most appropriate?

1. AA bar chart of the top 20 most frequent house_price values — converting the continuous price values into a ranked-frequency bar chart is the most accurate way to detect skewness or multimodality because it highlights the most common price points as discrete categories
2. BA pie chart showing the proportion of each unique house_price value out of the total dataset — pie charts reveal the relative frequency of each price point and scale well to thousands of unique continuous values, making them especially effective for visualizing the full distribution of high-cardinality numeric features
3. CA scatter plot of house_price against the row index in the DataFrame — plotting price values against their sequential index reveals the shape and spread of the continuous distribution and is the most appropriate univariate visualization tool for detecting skewness or multimodality
4. DA box-and-whisker plot only, without any histogram complement — box plots capture the median, interquartile range, and whisker bounds of the distribution and are the single most complete tool for detecting multimodality and the full shape of any continuous feature distribution
5. EA histogram that groups house_price values into bins and shows the count of observations per bin, created using df.plot.hist(column='house_price', bins=30) in PySpark native plotting, or by converting to Pandas and using df_pd['house_price'].hist(bins=30) — histograms directly visualize the shape of the distribution by aggregating continuous values into adjacent frequency bins, making it straightforward to identify skewness, outlier tails, and multiple modes

A data scientist is exploring a dataset with a product_category column containing 8 distinct nominal string values. She wants to understand the frequency distribution of each category. Which visualization type is most appropriate for this column?

1. AA line chart connecting the frequency count of each category in alphabetical order — line charts reveal trends in categorical frequencies over ordered discrete categories and are the standard approach for visualizing nominal categorical frequency distributions in exploratory data analysis
2. BA scatter plot with product_category on the x-axis and row index on the y-axis — scatter plots show where each category appears across the dataset rows, revealing unequal frequency distributions across nominal categories through the density of plotted points
3. CA histogram with automatic bin selection — histograms automatically detect nominal string columns and group them into equal-width bins by alphabetical ordering, making them the most versatile chart type for visualizing both categorical and continuous feature distributions in Databricks
4. DA bar chart showing the count of each category on the y-axis and category labels on the x-axis — created using Databricks built-in display() with bar chart visualization, or by calling df.groupBy('product_category').count().toPandas().plot.bar(), this chart directly shows each category's frequency as a proportional bar height, making it easy to compare category prevalences at a glance
5. EA box plot grouped by product_category — box plots display the five-number summary (min, Q1, median, Q3, max) for each category group and are the most appropriate chart type for understanding the frequency count distribution of a nominal categorical column in a Databricks notebook

A data scientist is building a PySpark MLlib preprocessing Pipeline for a dataset with three columns: annual_salary (double), years_experience (integer), and job_title (string). They plan to use PySpark's Imputer class to handle missing values across all three columns in one step. What is an important limitation they must account for?

1. AThe PySpark Imputer class only supports the 'mean' strategy and does not implement 'median' or 'mode' imputation — for any strategy other than mean, missing values must be manually computed using approxQuantile() or groupBy().agg() and applied with fillna() before constructing the Pipeline
2. BThe PySpark Imputer class requires that all missing values be encoded as Python None objects and will silently skip NaN floating-point values in double columns — all NaN values in numeric columns must first be converted to None using a custom UDF or F.when() expression before passing the DataFrame to the Imputer
3. CThe PySpark Imputer class only supports numeric data types such as integer and double — it does not support string or categorical columns, and the documentation explicitly warns that attempting to impute a categorical feature may produce incorrect imputed values; the job_title string column must be handled separately with a different imputation strategy before being passed to the Imputer
4. DThe PySpark Imputer class cannot be included as a stage inside a PySpark MLlib Pipeline object — it is a standalone utility that only functions correctly when called individually outside of a Pipeline, and placing it in a Pipeline stages list will raise an incompatible-stage runtime exception during Pipeline.fit()
5. EThe PySpark Imputer class automatically casts all double columns to integer before performing imputation if the computed surrogate value is a whole number — this automatic type casting can silently truncate decimal salary values to integers, resulting in loss of precision in the annual_salary column after imputation

A data scientist wants to explore a large Databricks Spark DataFrame to understand numeric distributions, string column frequencies, and see inline visual histograms of value distributions — all in a single command without writing extra plotting code. Which tool best satisfies all of these requirements inside a Databricks notebook?

1. ACall df.summary() and then pipe the resulting statistics DataFrame to matplotlib — df.summary() returns all statistical metrics and automatically renders distribution histograms for each column when its output is passed to matplotlib.show() in a Databricks notebook cell
2. BCall df.describe() — this PySpark method computes count, mean, stddev, min, and max for numeric and string columns and also renders embedded histograms alongside each metric when executed in a Databricks notebook environment
3. CUse df.stat.freqItems(['col1', 'col2']) — this method computes frequency distributions across all specified columns and automatically renders interactive visualizations for both numeric and categorical features directly in the Databricks notebook output cell
4. DCall dbutils.data.summarize(df) — this Databricks utility generates a rich data profile for the entire DataFrame including summary statistics, distinct value counts, and inline histograms of value distributions for numeric, string, and date columns, all rendered directly in the notebook
5. EConvert the Spark DataFrame to Pandas with .toPandas() and then call pandas_df.describe() — this produces the same comprehensive data profile as dbutils.data.summarize() and automatically renders inline histogram visualizations for each column without requiring additional imports or plotting code