Home / DE Professional practice test / Data Ingestion & Acquisition / Question

Databricks Data Engineer Professional · free practice question

A data engineering team ingests raw JSON and CSV drops landing every minute in a cloud object storage path. During peak hours, millions of small files arrive…

Exam-style question on Data Ingestion & Acquisition from the free DE Professional practice test, with the answer and a full explanation below.

Updated November 16, 2026

Data Ingestion & Acquisition Medium

A data engineering team ingests raw JSON and CSV drops landing every minute in a cloud object storage path. During peak hours, millions of small files arrive per hour, and the schema occasionally gains new columns. The pipeline must (1) process new files incrementally, (2) provide exactly-once processing guarantees when writing to Delta, and (3) avoid dropping unexpected columns. Which approach best fits these requirements with the least custom state management?

  1. ASchedule COPY INTO each minute with COPY_OPTIONS('force'='true') to reload files on each run; this picks up the latest data and prevents duplicates in the target Delta table.
  2. BUse spark.readStream.format("json").load(path) directly and rely on directory listing each trigger; store processed file names in a custom Delta table for exactly-once semantics.
  3. CUse Auto Loader with spark.readStream.format("cloudFiles"), a persistent checkpointLocation and cloudFiles.schemaLocation, keeping schema evolution on its default addNewColumns mode.
  4. DUse batch spark.read.format("csv").load(path) with recursiveFileLookup=true every minute and overwrite the target Delta table to avoid duplicates and handle schema drift.
Answer & explanation

Correct answer: C

WHY C: Auto Loader (cloudFiles, with cloudFiles.format set for the source files) is designed for incremental ingestion at scale, including very large file counts. It tracks progress in the streaming checkpoint and provides exactly-once guarantees when writing to Delta. With cloudFiles.schemaLocation (or Lakeflow Spark Declarative Pipelines managing these settings), it can infer and evolve the schema (default addNewColumns) or use rescue mode to retain unexpected fields in the rescued data column instead of silently dropping them.

WHY NOT B: The plain file source relies on directory listing and does not provide Auto Loader's scalable file discovery or built-in schema evolution/rescue patterns. A custom state table is extra operational burden and easy to get wrong.

WHY NOT A: COPY INTO is idempotent by default, but force=true disables that idempotency and reloads files, which causes duplicate ingestion. It also does not provide the streaming-style continuous file arrival handling of Auto Loader.

WHY NOT D: Overwriting the target on each run is not incremental processing: it re-reads the whole path, increases cost, and can introduce race conditions with concurrent readers. It is a heavy-handed way to handle duplicates and schema drift.

Take the full DE Professional practice test free →

Would you pass today?

This question is one of 60 in the free, timed Databricks Data Engineer Professional practice test. Take it and get a section-by-section score.

Start the free test →