Home / DE Professional practice test / Debugging and Deploying / Question

Databricks Data Engineer Professional · free practice question

A Lakeflow Spark Declarative Pipeline has a streaming flow ingesting from Apache Kafka but is falling progressively further behind, with the processing lag…

Exam-style question on Debugging and Deploying from the free DE Professional practice test, with the answer and a full explanation below.

Updated November 16, 2026

Debugging and Deploying Medium

A Lakeflow Spark Declarative Pipeline has a streaming flow ingesting from Apache Kafka but is falling progressively further behind, with the processing lag increasing each batch. The engineer wants to monitor streaming metrics such as input rate, processing rate, batch duration, and trigger interval FOR THE ACTIVE pipeline update in real time. Which tool surface should they use?

  1. AThe pipeline event log, queried with a filter for event_type = 'flow_progress' to get the latest batch statistics
  2. BThe system.lakeflow.pipeline_update_timeline system table, streamed in near-real-time using a Structured Streaming query
  3. CThe Monitor UI for the pipeline, which displays streaming source metrics including Kafka input rate, processing rate, and batch duration
  4. DThe cluster Metrics tab for the pipeline cluster, which shows active Spark tasks and Kafka consumer group offsets
Answer & explanation

Correct answer: C

WHY C is correct: The Databricks documentation explicitly states that the pipeline Monitor UI allows users to 'View metrics for streaming sources, like Apache Kafka and Auto Loader.' The Monitor UI shows real-time progress and status of pipeline updates including streaming-specific metrics such as input rate, processing rate, batch duration, and trigger interval. This is the designed, purpose-built surface for real-time monitoring of an active pipeline update and is the first place to look when diagnosing streaming pipeline lag.

WHY NOT A: The pipeline event log stores detailed update information and is the right tool for post-run analysis and data quality auditing. While it does record flow progress events that contain batch statistics, it is not a real-time streaming dashboard — it is a historical log that is queried after or during an update, not a live metrics display. The Monitor UI is the real-time observation surface.

WHY NOT B: system.lakeflow.pipeline_update_timeline records pipeline update-level metadata (update start time, end time, result state, triggered by). It tracks update lifecycle events, not intra-update streaming metrics like input rate per micro-batch or batch processing duration. Additionally, using Structured Streaming to query a Delta Sharing system table adds latency and complexity that is unnecessary when the Monitor UI is available.

WHY NOT D: The cluster Metrics tab shows Spark and hardware metrics for the cluster underlying the pipeline but does not expose streaming source-specific metrics like Kafka consumer lag, per-batch input row counts, or trigger intervals. Kafka consumer group offsets are tracked by the Kafka broker, not surfaced in the Databricks cluster Metrics tab.

Take the full DE Professional practice test free →

Would you pass today?

This question is one of 60 in the free, timed Databricks Data Engineer Professional practice test. Take it and get a section-by-section score.

Start the free test →