A data engineering team has an existing production Spark Structured Streaming job written in PySpark that reads from Delta Lake, applies a stateful sessionization window function using flatMapGroupsWithState, and writes results to a Gold Delta table. The team is evaluating migrating this job to Lakeflow Spark Declarative Pipelines. A senior engineer raises concerns about whether Lakeflow SDP can support all the features required. Which statement MOST accurately describes the capabilities and limitations of Lakeflow SDP compared to the team's direct Structured Streaming job for this specific use case?
Answer & explanation
Correct answer: A
WHY A: Lakeflow SDP is built on Spark Structured Streaming and handles the vast majority of streaming ETL patterns (APPLY CHANGES / AUTO CDC, aggregations, joins) extremely well, but it is a declarative framework that constrains table definitions to SQL-expressible or DataFrame API patterns that fit its table update model. flatMapGroupsWithState is a low-level stateful operator that manages arbitrary per-key state across micro-batches using a custom user-defined state transition function. This type of arbitrary stateful processing does not map cleanly to Streaming Table or Materialized View constructs. For such cases, direct Structured Streaming (the DataFrame API in a regular Python/Scala job) gives the team full access to every streaming operator. The guidance is to use Lakeflow SDP for the majority of pipeline logic and fall back to direct Structured Streaming when custom stateful operators are required.
WHY NOT B: @dlt.table functions do work with a streaming DataFrame, but the framework imposes constraints on which operators can be applied: arbitrary stateful operators like flatMapGroupsWithState are not supported within the declarative pipeline execution model.
WHY NOT C: Lakeflow SDP is not a replacement streaming engine; it runs on top of Spark Structured Streaming. It does not replace Structured Streaming internally, and flatMapGroupsWithState is not available within pipeline table definitions.
WHY NOT D: APPLY CHANGES INTO (AUTO CDC) is for CDC (insert/update/delete from a change stream), not for arbitrary sessionization window logic. The SEQUENCE BY clause determines event ordering for CDC operations, not custom stateful window computations.
WHY NOT E: Lakeflow SDP's management overhead does not add 15–30% query latency. The metadata and lineage operations are asynchronous and do not block query execution paths, and the two approaches are not equivalent in capability.