A data engineer has a Lakeflow Spark Declarative Pipelines pipeline where a streaming table clicks_bronze reads from a Delta table source source_clicks. To comply with GDPR, the engineer deletes rows from source_clicks using DML and simultaneously deletes the same rows from clicks_bronze. When the pipeline runs again, it fails with an error stating that the streaming source contains non-append operations. What change must the engineer make to clicks_bronze to resolve this without doing a full refresh?
Answer & explanation
Correct answer: B
WHY B: Streaming tables in Lakeflow Spark Declarative Pipelines process append-only data when they stream from Delta sources. When DML operations (DELETE, UPDATE) are applied to the source Delta table, the streaming read encounters non-append transactions and fails. The documented GDPR compliance pattern is to add .option('skipChangeCommits', 'true') to the readStream definition: the streaming table then skips non-append commits (deletes/updates) and continues processing new inserts, so the pipeline proceeds after DML-based deletions.
WHY NOT A: A materialized view does handle deletions from its source, but it is a different object type rather than a change to the streaming table, and it requires rewriting the pipeline definition. The question asks how to fix the streaming table without a full refresh.
WHY NOT C: Reading the change data feed exposes the delete events, which then need extra logic in the pipeline code to handle them; it does not simply skip them. It is a more complex redesign and not the recommended fix for this scenario.
WHY NOT D: Deleting the checkpoint and re-processing from the start is equivalent to a full refresh. It is expensive, can lose data if the upstream source has aged out data (e.g., Kafka with retention), and is not the targeted fix for the non-append commit problem.
WHY NOT E: retentionDurationCheck.enabled controls how short a VACUUM retention period can be set. It is unrelated to streaming table behavior or non-append commit handling.