A Structured Streaming job reads from Kafka using .option('startingOffsets', 'latest') and writes to a Delta sink with a checkpoint. After an incident, the job is redeployed and unexpectedly reprocesses old Kafka data from the beginning. Holding the Kafka topic and code constant, which change most directly prevents this behavior in the future?
Answer & explanation
Correct answer: D
WHY D: For Kafka Structured Streaming, the checkpoint is the durable source of truth for progress. startingOffsets is only used when starting a brand-new query; if the checkpoint is changed or deleted, the stream can start from the configured starting offsets (or earliest, depending on config) and reprocess old data. Keeping a stable checkpoint location is the direct fix.
WHY NOT A: endingOffsets is for batch reads, not a typical streaming configuration, and does not solve checkpoint loss.
WHY NOT C: Structured Streaming does not resume from Kafka consumer-group offsets. Setting kafka.group.id can cause interference between concurrently running queries using the same group ID and is recommended only with extreme caution; it is not a substitute for Structured Streaming checkpoints.
WHY NOT B: failOnDataLoss=false allows a query to continue despite missing Kafka data; it does not prevent replaying old offsets when checkpoint state is lost.