A bronze table may contain duplicate event rows that are identical across all columns due to at-least-once delivery. The silver job must keep exactly one copy of each fully-identical row. Which operation is correct and appropriate?
Answer & explanation
Correct answer: D
WHY D: With no subset, dropDuplicates() (same as distinct()) removes rows identical across all columns — exactly the requirement for fully-identical duplicates. WHY NOT A: Deduping on event_id alone collapses rows sharing an id but differing elsewhere, and dedup is not limited to one column. WHY NOT B: groupBy().count() returns aggregate counts, not the deduplicated row set in usable form. WHY NOT C: limit(1) keeps one row of the whole DataFrame, discarding real data. WHY NOT E: coalesce(1) only changes partitioning; it does not remove duplicate rows.