A classic job uses Auto Loader to ingest JSON files where new fields can appear without notice and some fields occasionally arrive with type mismatches. The pipeline must keep running (no manual restarts), and unexpected fields must not be silently dropped. Which configuration best satisfies these requirements while minimizing custom state handling?
Answer & explanation
Correct answer: A
WHY A: In rescue mode, Auto Loader does not fail the stream due to new columns and does not ignore unexpected fields; instead, it records unexpected columns and type/case mismatches in the rescued data column (default _rescued_data), allowing the pipeline to continue and letting you quarantine or inspect rescued content later. A persistent cloudFiles.schemaLocation and a stable checkpoint are required for reliable incremental processing and schema tracking, so no custom state handling is needed.
WHY NOT B: badRecordsPath stores malformed or corrupt records, but type mismatches and unexpected columns are handled by the rescued data column when it is configured; they are not necessarily treated as bad records, so this option confuses the corrupt-record sink with schema drift handling.
WHY NOT C: Removing the checkpoint breaks exactly-once guarantees and can cause reprocessing, and the checkpoint is what lets Auto Loader track state without custom code. addNewColumns also stops the stream when a new column arrives (it updates the schema and the stream typically needs a restart), and capturing mismatched values is the job of the rescued data column, not of a cast to strings.
WHY NOT D: When you provide an explicit schema, addNewColumns is not allowed in the same way (and defaults differ); you typically need schema hints or other handling. With an explicit schema and no rescued data column, mismatched values end up as nulls, which silently loses data the requirements say must be kept; this is not the keep-running configuration.