A Lakeflow Spark Declarative Pipeline has a streaming flow ingesting from Apache Kafka but is falling progressively further behind, with the processing lag increasing each batch. The engineer wants to monitor streaming metrics such as input rate, processing rate, batch duration, and trigger interval FOR THE ACTIVE pipeline update in real time. Which tool surface should they use?
Answer & explanation
Correct answer: C
WHY C is correct: The Databricks documentation explicitly states that the pipeline Monitor UI allows users to 'View metrics for streaming sources, like Apache Kafka and Auto Loader.' The Monitor UI shows real-time progress and status of pipeline updates including streaming-specific metrics such as input rate, processing rate, batch duration, and trigger interval. This is the designed, purpose-built surface for real-time monitoring of an active pipeline update and is the first place to look when diagnosing streaming pipeline lag.
WHY NOT A: The pipeline event log stores detailed update information and is the right tool for post-run analysis and data quality auditing. While it does record flow progress events that contain batch statistics, it is not a real-time streaming dashboard — it is a historical log that is queried after or during an update, not a live metrics display. The Monitor UI is the real-time observation surface.
WHY NOT B: system.lakeflow.pipeline_update_timeline records pipeline update-level metadata (update start time, end time, result state, triggered by). It tracks update lifecycle events, not intra-update streaming metrics like input rate per micro-batch or batch processing duration. Additionally, using Structured Streaming to query a Delta Sharing system table adds latency and complexity that is unnecessary when the Monitor UI is available.
WHY NOT D: The cluster Metrics tab shows Spark and hardware metrics for the cluster underlying the pipeline but does not expose streaming source-specific metrics like Kafka consumer lag, per-batch input row counts, or trigger intervals. Kafka consumer group offsets are tracked by the Kafka broker, not surfaced in the Databricks cluster Metrics tab.