A data engineer sets up an Auto Loader stream with spark.readStream.format('cloudFiles') to ingest JSON files as they land in cloud storage. Which configuration lets Auto Loader process each incoming file exactly once, even across stream restarts?
Answer & explanation
Correct answer: A
WHY A: Auto Loader is built on Structured Streaming and requires a checkpoint location plus a source format; it tracks processed files in that checkpoint (RocksDB), giving exactly-once ingestion across restarts.
WHY NOT B: There is no force option in Auto Loader (that belongs to COPY INTO), and exactly-once comes from file tracking, not checksum comparison.
WHY NOT C: File state lives in the checkpoint location you supply, not in the target table's transaction log.
WHY NOT D: AvailableNow controls when processing stops, not the exactly-once guarantee, which comes from checkpointing under any trigger.
WHY NOT E: inferColumnTypes and the schema location handle schema, not the processed-file registry, which is the checkpoint's job.