A streaming pipeline ingests events and uses foreachBatch to write the same micro-batch into two Delta tables (a raw table and an audit table). During intermittent failures, the job retries a batch and occasionally writes duplicates. Which configuration best makes the batch writes idempotent so retries do not duplicate data?
Answer & explanation
Correct answer: C
WHY C: Delta supports idempotent writes using the (txnAppId, txnVersion) pair. When a batch is retried, reusing the same app ID and batch ID lets Delta identify duplicate writes for that pair and skip them. This is especially important when doing multiple writes inside foreachBatch.
WHY NOT A: Checkpoints are required for Structured Streaming correctness; Delta does not deduplicate rows by itself, and removing checkpoints makes reprocessing more likely.
WHY NOT B: COPY INTO is for loading from file locations; force=true disables its idempotency and can introduce duplicates.
WHY NOT D: Schema evolution settings do not address at-least-once retry behavior; duplicates are a write idempotency problem, not a schema problem.