Home / DE Professional practice test / Developing Code for Data Processing using Python and SQL / Question

Databricks Data Engineer Professional · free practice question

A data engineering team is building a production pipeline that ingests raw JSON event files landing continuously in an Azure Data Lake Storage Gen2 container…

Exam-style question on Developing Code for Data Processing using Python and SQL from the free DE Professional practice test, with the answer and a full explanation below.

Updated November 16, 2026

Developing Code for Data Processing using Python and SQL Medium

A data engineering team is building a production pipeline that ingests raw JSON event files landing continuously in an Azure Data Lake Storage Gen2 container. The pipeline must: (1) detect new files as they arrive with sub-minute latency, (2) handle schema evolution gracefully when new fields are added to the JSON payload, (3) track which files have been processed to avoid reprocessing on pipeline restart, (4) scale to handle bursts of thousands of files arriving within minutes. The team is choosing between configuring Auto Loader with file notification mode vs. directory listing mode. Which configuration correctly addresses all four requirements?

  1. AUse Auto Loader with directory listing mode (cloudFiles.useNotifications=false). Directory listing is more reliable for ADLS Gen2 because file notification events from Azure Event Grid can arrive out of order, causing Auto Loader to miss files. Directory listing checks for all new files on each trigger, eliminating any risk of missed events. Set cloudFiles.schemaEvolutionMode=rescue to capture new fields without failing.
  2. BUse a standard Structured Streaming readStream with format('json') and the maxFilesPerTrigger option set to 1000. This approach uses directory listing to find new files and is preferable to Auto Loader for large bursts of files because the native JSON streaming reader has lower metadata overhead than Auto Loader's cloudFiles format. Schema evolution is handled by setting mode=PERMISSIVE in the JSON read options.
  3. CUse Auto Loader with file notification mode (cloudFiles.useNotifications=true), which configures Azure Event Grid and Azure Queue Storage to deliver real-time file arrival events to Auto Loader. This provides sub-minute detection latency at any scale. Set cloudFiles.schemaEvolutionMode=addNewColumns so newly arriving JSON fields are automatically added to the inferred schema. Auto Loader's checkpoint mechanism tracks processed files in the streaming checkpoint location, preventing reprocessing on restart.
  4. DUse Auto Loader in directory listing mode with cloudFiles.backfillInterval=1 minute to check for new files every minute. Schema evolution is not natively supported by Auto Loader for JSON sources; instead, use from_json(col('_raw'), schema) to parse the JSON payload from a raw string column, providing a fixed schema that never needs to evolve. The fixed schema approach is more performant than dynamic schema inference for production pipelines.
  5. EUse a Lakeflow Spark Declarative Pipeline with a STREAMING TABLE source backed by Auto Loader in file notification mode. Configure the pipeline in CONTINUOUS mode to achieve sub-minute latency. For schema evolution, set cloudFiles.schemaEvolutionMode=failOnNewColumns so that the pipeline fails immediately when unexpected fields arrive, triggering an alert to the team to manually update the schema before resuming — this prevents silent schema drift in production.
Answer & explanation

Correct answer: C

WHY C: File notification mode with Azure Event Grid + Azure Queue Storage is the correct choice for sub-minute latency at scale. Notifications are pushed in near-real-time as files land, unlike directory listing which must periodically scan the entire prefix. addNewColumns schema evolution mode automatically extends the inferred schema when new JSON fields appear, satisfying requirement 2. Auto Loader's streaming checkpoint durably records processed file offsets, preventing reprocessing on restart (requirement 3). Event Grid + Queue Storage scales to handle thousands of events per second, satisfying requirement 4. WHY NOT A: Directory listing mode scans the directory on each trigger interval, which introduces latency proportional to the directory size and does not provide sub-minute detection at scale. rescue mode captures unknown fields in a _rescued_data column but does not add them to the schema. WHY NOT B: Standard readStream with format('json') is essentially what Auto Loader uses internally, but without the file tracking checkpoint, notification integration, or schema inference capabilities. PERMISSIVE mode captures malformed records but does not handle schema evolution. WHY NOT D: backfillInterval is for catching up on missed files, not for real-time detection. Auto Loader fully supports JSON schema evolution natively — cloudFiles.schemaEvolutionMode is specifically designed for this. WHY NOT E: failOnNewColumns would cause the pipeline to fail every time a new JSON field is added, which is the opposite of graceful schema evolution and creates operational burden.

Take the full DE Professional practice test free →

Would you pass today?

This question is one of 60 in the free, timed Databricks Data Engineer Professional practice test. Take it and get a section-by-section score.

Start the free test →