An engineering group intends to establish a Databricks pipeline for incrementally processing text chunks. Daily additions of PDF documents occur within a Unity Catalog Volume. This pipeline is required to: (1) identify solely the newly appeared PDFs in each execution cycle, (2) extract and segment their textual content, (3) add new chunk entries to an established catalog.rag.chunks_bronze Delta table without disturbing existing records, and (4) ensure no duplicate chunks are created if a document is inadvertently processed more than once. What sequence of operations would accurately satisfy these four conditions?
Answer & explanation
Correct answer: B
WHY B is correct: A processed_files tracking table with a LEFT ANTI JOIN identifies only the PDFs not yet handled, and writing the chunks with MERGE INTO on the (doc_id, chunk_index) key adds new rows without touching existing ones while updating matches instead of inserting duplicates when a document is reprocessed. Together these steps cover all four requirements.
WHY NOT E: Delta Lake has no feature that detects and drops identical rows on append, so reprocessing a document would insert its chunks a second time. A side text file is also a fragile way to track which files were handled.
WHY NOT A: Auto Loader's checkpoint does handle incremental file discovery, but this sequence appends the chunks to the target table before merging. If a document is processed again, the append has already inserted the duplicate rows, and the later MERGE does not remove them, so the fourth requirement is not met.
WHY NOT C: Overwriting the table on every run violates the requirement to leave existing records untouched and reprocesses the whole corpus each time, which is not incremental. Time travel allows recovery but does not make an overwrite an incremental load.
WHY NOT D: ACID guarantees make each write atomic and isolate concurrent transactions, but they do not stop a later run from appending the same rows again. Plain append mode has no built-in deduplication.